APPARATUS AND METHOD FOR PERFORMING ARTIFICIAL INTELLIGENCE (Al) ENCODING AND Al DECODING ON IMAGE

ABSTRACT

An artificial intelligence (AI) decoding method including obtaining image data generated from performing first encoding on a first image and AI data related to AI down-scaling of at least one original image related to the first image; obtaining a second image corresponding to the first image by performing first decoding on the image data; obtaining, based on the AI data, deep neural network (DNN) setting information for performing AI up-scaling of the second image; and generating a third image by performing the AI up-scaling on the second image via an up-scaling DNN operating according to the obtained DNN setting information. The DNN setting information is DNN information updated for performing the AI up-scaling of at least one second image via joint training of the up-scaling DNN and a down-scaling DNN used for the AI down-scaling.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a Continuation of U.S. application Ser. No.17/180,358, filed on Feb. 19, 2021, which is a Continuation of U.S.application Ser. No. 17/082,442 filed on Oct. 28, 2020, in the U.S.Patent and Trademark Office, which claims priority under 35 U.S.C. § 119to Korean Patent Application No. 10-2019-0134514, filed on Oct. 28,2019, in the Korean Intellectual Property Office, the disclosures ofwhich are herein incorporated by reference in their entireties.

BACKGROUND 1. Field

The disclosure relates to the field of image processing and, moreparticularly, to methods and apparatuses for encoding and decoding animage, based on artificial intelligence (AI).

2. Description of Related Art

An image is encoded using a codec conforming to a predefined datacompression standard such as a Moving Picture Expert Group (MPEG)standard, and then is stored in a recording medium or transmittedthrough a communication channel in the form of a bitstream.

With the development and dissemination of hardware capable ofreproducing and storing high-resolution/high-definition images, there isan increasing need for a codec capable of effectively encoding anddecoding high-resolution/high-definition images.

SUMMARY

According to embodiments of the disclosure, there are provided methodsand apparatuses for performing artificial intelligence (AI) encoding andAI decoding on an image, whereby a low bitrate may be achieved byencoding and decoding the image based on AI.

Also, according to embodiments of the disclosure, there are provided amethod and apparatus for performing AI encoding and AI decoding on animage, whereby image quality may be improved by performing up-scalingafter updating, periodically or whenever necessary, up-scaling DNNsetting information optimized for an original image.

Also, according to embodiments of the disclosure, there are provided amethod and apparatus for performing AI encoding and AI decoding on animage, whereby the amount of information to be encoded/decoded may beeffectively reduced by effectively signaling DNN setting information forupdating DNN setting information of an up-scaling DNN, optimized for anoriginal image, periodically or whenever necessary.

Additional aspects will be set forth in part in the description whichfollows and, in part, will be apparent from the description, or may belearned by practice of the presented embodiments of the disclosure.

According to an embodiment of the disclosure, an AI decoding apparatusincludes: a memory storing one or more instructions; and a processorconfigured to execute the one or more instructions stored in the memoryto: obtain image data generated from performing first encoding on afirst image and AI data related to AI down-scaling of at least oneoriginal image related to the first image; obtain a second imagecorresponding to the first image by performing first decoding on theimage data; obtain, based on the AI data, DNN setting information forperforming AI up-scaling of the second image; and generate a third imageby performing the AI up-scaling on the second image via an up-scalingDNN operating according to the obtained DNN setting information, whereinthe DNN setting information is DNN information updated for performingthe AI up-scaling of at least one second image corresponding to the atleast one original image via joint training of the up-scaling DNN and adown-scaling DNN used for the AI down-scaling of the at least oneoriginal image, the joint training being performed using the at leastone original image.

The obtained DNN setting information may include weights and biases offilter kernels in at least one convolution layer of the up-scaling DNN.

The processor may be further configured to generate a first trainingimage via the down-scaling DNN by using the at least one original image,generate a second training image via the up-scaling DNN by using thefirst training image, and update the up-scaling DNN and the down-scalingDNN based on first loss information and third loss information, thefirst loss information and the third loss information corresponding to aresult of comparing the second training image with an original imagethat has not undergone the AI down-scaling among the at least oneoriginal image, and second loss information generated based on the firsttraining image.

The first loss information may be generated based on a result ofcomparing a quality parameter of the second training image with aquality parameter of the at least one original image.

The third loss information may be generated based on a result ofcomparing a feature-related parameter of the second training image witha feature-related parameter of the at least one original image.

The second loss information may be related to a spatial complexity ofthe first training image.

The processor may be further configured to generate a first trainingimage via the down-scaling DNN by using the at least one original image,perform first encoding on the first training image, generate a secondtraining image via the up-scaling DNN by using the first training imagethat has undergone the first encoding, and update the up-scaling DNNbased on first loss information and third loss information, the firstloss information and the third loss information corresponding to aresult of comparing the second training image with an original imagethat has not undergone the AI down-scaling among the at least oneoriginal image.

The updated DNN setting information of the up-scaling DNN may includeweight residual information/bias residual information indicating adifference between a weight/a bias of all or some of filter kernels inall or some of convolution layers in the up-scaling DNN before theweight/the bias are updated and a weight/a bias of the all or some ofthe filter kernels in the all or some of the convolution layers in theup-scaling DNN after the weight/the bias are updated.

The updated DNN setting information of the up-scaling DNN may includeinformation about a weight residual/a bias residual obtained byperforming frequency transformation, the information about the weightresidual/the bias residual indicating a difference between a weight/abias of all or some of filter kernels in all or some of convolutionlayers in the up-scaling DNN before the weight/the bias are updated anda weight/a bias of the all or some of the filter kernels in the all orsome of the convolution layers in the up-scaling DNN after theweight/the bias are updated.

The weight residual information/bias residual information may beinformation encoded using one of differential pulse code modulation(DPCM), run-length coding (RLC), and Huffman coding techniques.

The weight residual information/bias residual information may beinformation about a weight residual/a bias residual generated via modelcompression.

The model compression may include at least one of pruning orquantization.

The updated DNN setting information of the up-scaling DNN may beinformation updated for performing the AI up-scaling obtained byentropy-encoding a weight/a bias of all or some of filter kernels in allor some of convolution layers in the up-scaling DNN after the weight/thebias are updated, based on context model information regarding aweight/a bias of the all or some of the filter kernels in the all orsome of the convolution layers in the up-scaling DNN before theweight/the bias are updated.

The DNN setting information may include flag information indicatingwhether to perform the AI up-scaling by using a filter kernel of aconvolution layer in a predetermined DNN or whether to perform the AIup-scaling by using a filter kernel of a convolution layer in a DNNupdated for performing the AI up-scaling of the at least one secondimage corresponding to the at least one original image via jointtraining of the up-scaling DNN and the down-scaling DNN used for the AIdown-scaling of the at least one original image, the joint trainingbeing performed using the at least one original image.

According to another embodiment of the disclosure, an AI encodingapparatus includes: a memory storing one or more instructions; and aprocessor configured to execute the one or more instructions stored inthe memory to: obtain a first image by performing AI down-scaling on atleast one original image via a down-scaling DNN; generate AI encodingdata comprising the image data and AI data including information relatedto the AI down-scaling and DNN setting information of an up-scaling DNNfor performing AI up-scaling on a second image, wherein the second imageis generated by performing first decoding on the image data, and thesecond image is generated by performing first decoding on the imagedata, and wherein the DNN setting information is DNN information updatedfor performing the AI up-scaling of at least one second imagecorresponding to the at least one original image via joint training ofthe up-scaling DNN and the down-scaling DNN used for the AI down-scalingof the at least one original image, the joint training being performedusing the at least one original image.

According to another embodiment of the disclosure, an AI decoding methodincludes: obtaining image data generated from performing first encodingon a first image and AI data related to AI down-scaling of at least oneoriginal image related to the first image; obtaining a second imagecorresponding to the first image by performing first decoding on theimage data; obtaining, based on the AI data, DNN setting information forperforming AI up-scaling of the second image; and generating a thirdimage by performing the AI up-scaling on the second image via anup-scaling DNN operating according to the obtained DNN settinginformation, wherein the DNN setting information is DNN informationupdated for performing the AI up-scaling of at least one second imagecorresponding to the at least one original image via joint training ofthe up-scaling DNN and a down-scaling DNN used for the AI down-scalingof the at least one original image, the joint training being performedusing the at least one original image.

According to another embodiment of the disclosure, an AI encoding methodincludes: obtaining a first image by performing AI down-scaling on atleast one original image via a down-scaling DNN; generating image databy performing first encoding on the first image; and generating AIencoding data comprising the image data and AI data includinginformation related to the AI down-scaling and DNN setting informationof an up-scaling DNN for performing AI up-scaling on a second image,wherein the second image is generated by performing first decoding onthe image data, and wherein the DNN setting information is DNNinformation updated for performing the AI up-scaling of at least onesecond image corresponding to the at least one original image via jointtraining of the up-scaling DNN and the down-scaling DNN used for the AIdown-scaling of the at least one original image, the joint trainingbeing performed using the at least one original image.

According to another embodiment of the disclosure, a computer-readablerecording medium has recorded thereon a program for executing theabove-described AI decoding method and AI encoding method.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certainembodiments of the disclosure will be more apparent from the followingdescription taken in conjunction with the accompanying drawings, inwhich:

FIG. 1 is a diagram for describing an artificial intelligence (AI)encoding process and an AI decoding process, according to an embodiment;

FIG. 2 is a block diagram of an AI decoding apparatus according to anembodiment;

FIG. 3 is a diagram showing a deep neural network (DNN) for performingAI up-scaling on an image;

FIG. 4 is a diagram for describing a convolution operation by aconvolution layer;

FIG. 5 is a table showing a mapping relationship between image-relatedinformation and DNN setting information;

FIG. 6 is a diagram showing an image including a plurality of frames;

FIG. 7 is a block diagram of an AI encoding apparatus according to anembodiment;

FIG. 8 is a diagram showing a DNN for performing AI down-scaling on anoriginal image;

FIG. 9 is a diagram for describing a method of training a first DNN anda second DNN;

FIG. 10 is a diagram for describing a training process of a first DNNand a second DNN by a training apparatus;

FIG. 11 is a diagram of an apparatus for performing AI down-scaling onan original image and an apparatus for performing AI up-scaling on asecond image;

FIG. 11 is a diagram illustrating an apparatus for performing AIdown-scaling on an original image and an apparatus for performing AIup-scaling on a second image;

FIG. 12 is a block diagram of a configuration of an AI encodingapparatus according to an embodiment of the disclosure;

FIG. 13 is a block diagram of a configuration of an AI decodingapparatus according to an embodiment of the disclosure;

FIG. 14A is a flowchart of an AI encoding method according to anembodiment of the disclosure;

FIG. 14B is a flowchart of an AI encoding method via DNN joint trainingbased on an original image, according to an embodiment of thedisclosure;

FIG. 14C is a flowchart of an AI encoding method via DNN separatetraining based on an original image, according to an embodiment of thedisclosure;

FIG. 15 is a flowchart of an AI decoding method according to anembodiment of the disclosure;

FIG. 16A is a diagram for describing, as a first stage of jointtraining, a process, performed by an AI encoding apparatus, ofdetermining pieces of optimal setting information of a down-scaling DNNand an up-scaling DNN via joint training of the down-scaling DNN and theup-scaling DNN by using an original image as a training image, accordingto an embodiment of the disclosure;

FIG. 16B is a diagram for describing, as a second stage of jointtraining, a process, performed by an AI encoding apparatus, of selectingthe setting information of the down-scaling DNN, which is determinedaccording to the process illustrated in FIG. 16A, determining optimalsetting information of an up-scaling DNN via separate training of theup-scaling DNN, and generating AI encoding data including optimalsetting information of the up-scaling DNN, according to an embodiment ofthe disclosure;

FIG. 16C is a diagram of a process, performed by an AI decodingapparatus, of performing AI up-scaling on a second image via anup-scaling DNN based on setting information of the up-scaling DNN, whichis included in AI encoding data, according to an embodiment of thedisclosure;

FIG. 17A is a diagram for describing a process, performed by an AIencoding apparatus, of selecting setting information of a down-scalingDNN, determining optimal DNN setting information of an up-scaling DNNvia separate training of the up-scaling DNN, and generating AI encodingdata including optimal DNN setting information of the up-scaling DNN,according to an embodiment of the disclosure;

FIG. 17B is a diagram for describing a process, performed by an AIdecoding apparatus, of performing AI up-scaling on a second image via anup-scaling DNN based on DNN setting information of the up-scaling DNN,which is included in AI encoding data, according to an embodiment of thedisclosure;

FIG. 18 is a flowchart of a process, performed by an AI decodingapparatus, of up-scaling a second image by updating DNN settinginformation of an up-scaling DNN, which is predetermined based on flagsobtained from AI encoding data or of up-scaling the second image byupdating DNN setting information of the up-scaling DNN, which isoptimized for an original image, according to an embodiment of thedisclosure;

FIG. 19A illustrates examples of default weights and biases, weights andbiases in an up-scaling DNN, which are optimized for an original image,and weight differences and bias differences in the up-scaling DNN,according to an embodiment of the disclosure;

FIG. 19B illustrates examples of weights and biases of an up-scalingDNN, which are optimized for an original image, and weights and biasesin the up-scaling DNN, which are determined via quantization andpruning, according to an embodiment of the disclosure;

FIG. 20A is a diagram for describing a method of encoding weights in anup-scaling DNN, which are optimized for an original image, according toan embodiment of the disclosure;

FIG. 20B is a diagram for describing a method of encoding weights in anup-scaling DNN, which are optimized for an original image, according toan embodiment of the disclosure;

FIG. 21A is a diagram for describing a process, performed by an AIencoding apparatus, of entropy-encoding weight information of anup-scaling DNN, which is optimized for an original image, based on acontext model, according to an embodiment of the disclosure; and

FIG. 21B is a diagram for describing a process, performed by an AIdecoding apparatus, of entropy-decoding weight information of theup-scaling DNN, which is optimized for the original image, based on thecontext model, according to an embodiment of the disclosure.

DETAILED DESCRIPTION

As the disclosure allows for various changes and numerous examples,particular embodiments will be illustrated in the drawings and describedin detail in the written description. However, this is not intended tolimit the disclosure to particular modes of practice, and it will beunderstood that all changes, equivalents, and substitutes that do notdepart from the spirit and technical scope of the disclosure areencompassed in the disclosure.

In the description of the embodiments, certain detailed explanations ofrelated art are omitted when such explanations might unnecessarilyobscure the essence of the disclosure. Also, numbers (for example, afirst, a second, and the like) used in the description of thespecification are merely identifying labels for distinguishing oneelement from another.

Throughout the disclosure, the expression “at least one of a, b or c”includes only a, only b, only c, both a and b, both a and c, both b andc, all of a, b, and c, or variations thereof.

Also, in the disclosure, it will be understood that when elements are“connected” or “coupled” to each other, the elements may be directlyconnected or coupled to each other, but may alternatively be connectedor coupled to each other with an intervening element therebetween,unless specified otherwise.

In the disclosure, regarding an element represented as a “unit” or a“module,” two or more elements may be combined into one element or oneelement may be divided into two or more elements according to subdividedfunctions. In addition, each element described hereinafter mayadditionally perform some or all of functions performed by anotherelement, in addition to main functions of itself, and some of the mainfunctions of each element may be performed entirely by anothercomponent.

Also, in the disclosure, an ‘image’ or a ‘picture’ may denote a stillimage, a moving image including a plurality of consecutive still images(or frames), or a video.

Also, in the disclosure, a deep neural network (DNN) is a representativeexample of an artificial neural network model simulating brain nerves,and is not limited to an artificial neural network model using aspecific algorithm.

Also, in the disclosure, a ‘parameter’ is a value used in an operationprocess of each layer forming a neural network, and for example, mayinclude a weight used when an input value is applied to a certainoperation expression. Here, the parameter may be expressed in a matrixform. The parameter is a value set as a result of training or iterationof the neural network, and may be updated through separate training datawhen necessary.

Also, in the disclosure, a ‘first DNN’ indicates a DNN used forartificial intelligence (AI) down-scaling an image, and a ‘second DNN’indicates a DNN used for AI up-scaling an image.

Also, in the disclosure, ‘DNN setting information’ includes informationrelated to an element constituting a DNN. ‘DNN setting information’includes the parameter described above as information related to theelement constituting the DNN. The first DNN or the second DNN may be setby using the DNN setting information.

Also, in the disclosure, an ‘original image’ denotes an image that is anobject of AI encoding, and a ‘first image’ denotes an image obtained asa result of performing AI down-scaling on the original image during anAI encoding process. Also, a ‘second image’ denotes an image obtainedvia first decoding during an AI decoding process, and a ‘third image’denotes an image obtained by AI up-scaling the second image during theAI decoding process.

Also, in the disclosure, ‘AI down-scale’ (or downscaling) denotes aprocess of decreasing a resolution of an image based on AI, and ‘firstencoding’ denotes an encoding process according to an image compressionmethod based on frequency transformation. Also, ‘first decoding’ denotesa decoding process according to an image reconstruction method based onfrequency transformation, and ‘AI up-scale’ (or upscaling) denotes aprocess of increasing a resolution of an image based on AI.

FIG. 1 is a diagram for describing an AI encoding process and an AIdecoding process, according to an embodiment.

When resolution of an image remarkably increases, the throughput ofinformation for encoding and decoding the image is increased.Accordingly, a method for improving the efficiency of encoding anddecoding of an image is required.

As shown in FIG. 1, according to an embodiment of the disclosure, afirst image 115 is obtained by performing AI down-scaling 110 on anoriginal image 105 having high resolution. Then, first encoding 120 andfirst decoding 130 are performed on the first image 115 havingrelatively low resolution, and thus a bitrate may be significantlyreduced compared to simply performing the first encoding and the firstdecoding on the high resolution original image 105.

In particular, in FIG. 1, the first image 115 is obtained by performingthe AI down-scaling 110 on the original image 105 and the first encoding120 is performed by an encoding apparatus, transmitting source, or thelike on the first image 115 during the AI encoding process, according toan embodiment. During the AI decoding process, AI encoding dataincluding AI data and image data, which are obtained as a result of AIencoding is received, a second image 135 is obtained via the firstdecoding 130, and a third image 145 is obtained by a decoding apparatus,receiving device, or the like performing AI up-scaling 140 on the secondimage 135.

Referring to the AI encoding process in detail, when the original image105 is received, the AI down-scaling 110 is performed on the originalimage 105 to obtain the first image 115 of certain resolution or certainquality. Here, the AI down-scaling 110 is performed based on AI, and theAI model for performing the AI down-scaling 110 needs to be trainedjointly with the AI model for performing the AI up-scaling 140 of thesecond image 135. This is because, when the AI model for the AIdown-scaling 110 and the AI model for the AI up-scaling 140 areseparately trained, there may exist distortion due to a differencebetween the original image 105 subjected to AI encoding and the thirdimage 145 reconstructed through AI decoding.

In an embodiment of the disclosure, the AI data may be signaling used tomaintain such a joint relationship during the AI encoding process andthe AI decoding process. Accordingly, the AI data obtained through theAI encoding process may include information indicating an up-scalingtarget, and during the AI decoding process, the AI up-scaling 140 isperformed on the second image 135 according to the up-scaling targetverified based on the AI data.

The AI model for the AI down-scaling 110 and the AI model for the AIup-scaling 140 may be embodied as a DNN. As will be described later withreference to FIG. 9, because a first DNN and a second DNN are jointlytrained by sharing loss information under a certain target, an AIencoding apparatus may provide target information used during jointtraining of the first DNN and the second DNN to an AI decodingapparatus, and the AI decoding apparatus may perform the AI up-scaling140 on the second image 135 to target resolution based on the providedtarget information.

Regarding the first encoding 120 and the first decoding 130 of FIG. 1,an information amount of the first image 115 obtained by performing AIdown-scaling 110 on the original image 105 may be reduced through thefirst encoding 120. The first encoding 120 may include: a process ofgenerating prediction data by predicting the first image 115, a processof generating residual data corresponding to a difference between thefirst image 115 and the prediction data, a process of transforming theresidual data of a spatial domain component to a frequency domaincomponent, a process of quantizing the residual data transformed to thefrequency domain component, and a process of entropy-encoding thequantized residual data. Such first encoding 120 may be performed viaone of image compression methods using frequency transformation, such asMPEG-2, H.264 Advanced Video Coding (AVC), MPEG-4, H.265/High EfficiencyVideo Coding (HEVC), VC-1, VP8, VP9, and AOMedia Video 1 (AV1).

The second image 135 corresponding to the first image 115 may bereconstructed by performing the first decoding 130 on the image data.The first decoding 130 may include: a process of generating thequantized residual data by entropy-decoding the image data, a process ofinverse-quantizing the quantized residual data, a process oftransforming the residual data of the frequency domain component to thespatial domain component, a process of generating the prediction data,and a process of reconstructing the second image 135 by using theprediction data and the residual data. Such first decoding 130 may beperformed via a corresponding image reconstruction method correspondingto one of image compression methods using frequency transformation, suchas MPEG-2, H.264 AVC, MPEG-4, H.265/HEVC, VC-1, VP8, VP9, and AV1, whichis used in the first encoding 120.

The AI encoding data obtained through the AI encoding process mayinclude the image data obtained as a result of performing the firstencoding 120 on the first image 115, and the AI data related to the AIdown-scaling 110 of the original image 105. The image data may be usedduring the first decoding 130 and the AI data may be used during the AIup-scaling 140.

The image data may be transmitted in a form of a bitstream. The imagedata may include data obtained based on pixel values in the first image115, for example, residual data that is a difference between the firstimage 115 and prediction data of the first image 115. Also, the imagedata includes information used during the first encoding 120 performedon the first image 115. For example, the image data may includeprediction mode information, motion information, and information relatedto a quantization parameter used during the first encoding 120. Theimage data may be generated according to a rule, for example, accordingto a syntax, of an image compression method used during the firstencoding 120, among MPEG-2, H.264 AVC, MPEG-4, H.265/HEVC, VC-1, VP8,VP9, and AV1.

The AI data is used in the AI up-scaling 140 based on the second DNN. Asdescribed above, because the first DNN and the second DNN are jointlytrained, the AI data includes information enabling the AI up-scaling 140to be consistently performed on the second image 135 through the secondDNN. During the AI decoding process, the AI up-scaling 140 may beperformed on the second image 135 to have targeted resolution and/orquality, based on the AI data.

The AI data may be transmitted together with the image data in a form ofa bitstream. Alternatively, according to an embodiment, the AI data maybe transmitted separately from the image data, in a form of a frame or apacket. The AI data and the image data obtained as a result of the AIencoding may be transmitted through the same network or throughdifferent networks.

FIG. 2 is a block diagram of an AI decoding apparatus according to anembodiment.

Referring to FIG. 2, the AI decoding apparatus 200 according to anembodiment may include a receiver 210 and an AI decoder 230. Thereceiver 210 may include a communicator 212, a parser 214, and anoutputter 216. The AI decoder 230 may include a first decoder 232 and anAI up-scaler 234.

The receiver 210 receives and parses AI encoding data obtained as aresult of AI encoding, and outputs image data and AI data to the AIdecoder 230.

In particular, the communicator 212 receives the AI encoding dataobtained as the result of AI encoding through a wireless network. The AIencoding data obtained as the result of performing AI encoding includesthe image data and the AI data. The image data and the AI data may bereceived through a same type of network or different types of networks.

The parser 214 receives the AI encoding data received through thecommunicator 212 and parses the AI encoding data to segment the imagedata from the AI data. For example, the parser 214 may distinguish theimage data and the AI data by reading a header of data obtained from thecommunicator 212 that describes the payload including the image data andthe AI data or a configuration of the AI data. According to anembodiment, the parser 214 transmits the image data and the AI data tothe outputter 216 via the header of the data received through thecommunicator 212, and the outputter 216 transmits the image data and AIdata respectively to the first decoder 232 and the AI up-scaler 234. Atthis time, the image data included in the AI encoding data may beverified to be image data generated via a certain codec (for example,MPEG-2, H.264 AVC, MPEG-4, HEVC, VC-1, VP8, VP9, or AV1). In this case,corresponding information may be transmitted to the first decoder 232through the outputter 216 such that the image data is processed via theappropriate codec.

According to an embodiment, the AI encoding data parsed by the parser214 may be obtained from a data storage medium including a magneticmedium such as a hard disk, a floppy disk, or a magnetic tape, anoptical recording medium such as CD-ROM or DVD, or a magneto-opticalmedium such as a floptical disk.

The first decoder 232 reconstructs the second image 135 corresponding tothe first image 115, based on the image data. The second image 135obtained by the first decoder 232 is provided to the AI up-scaler 234.According to an embodiment, first decoding related information, such asprediction mode information, motion information, quantization parameterinformation, or the like included in the image data may be furtherprovided to the AI up-scaler 234.

Upon receiving the AI data, the AI up-scaler 234 performs AI up-scalingon the second image 135, based on the AI data. According to anembodiment, the AI up-scaling may be performed by further using thefirst decoding related information, such as the prediction modeinformation, the quantization parameter information, or the likeincluded in the image data.

The receiver 210 and the AI decoder 230 according to an embodiment aredescribed as individual components or devices, but may be jointlyimplemented through one processor for controlling the jointfunctionality of the receiver 210 and the decoder 230. In this case, thereceiver 210 and the AI decoder 230 may be implemented through adedicated processor or through a combination of software and ageneral-purpose processor such as application processor (AP), centralprocessor (CPU) or graphic processor (GPU). The dedicated processor maybe implemented by including a memory for implementing an embodiment ofthe disclosure or by including a memory processor for using an externalmemory. The communicator 212 may include a communication interface, suchas a wireless communication interface including a radio and/or anantenna or a wired communication interface such as HDMI, Ethernet, andthe like, for receiving the AI encoding data over a network or from adata storage medium including a magnetic medium such as a hard disk, afloppy disk, or a magnetic tape, an optical recording medium such asCD-ROM or DVD, or a magneto-optical medium such as a floptical disk.Alternatively, the communicator 212 may include a portion or componentof a dedicated processor or through a combination of software and ageneral-purpose processor such as application processor (AP), centralprocessor (CPU) or graphic processor (GPU) for receiving the AI encodingdata.

Also, the receiver 210 and the AI decoder 230 may be configured by aplurality of processors. In this case, the receiver 210 and the AIdecoder 230 may be implemented through a combination of dedicatedprocessors or through a combination of software and general-purposeprocessors such as AP, CPU or GPU. Similarly, the AI up-scaler 234 andthe first decoder 232 may be implemented by different processors.

The AI data provided to the AI up-scaler 234 includes informationenabling the second image 135 to be processed via AI up-scaling. Here,an up-scaling target or ratio should correspond to down-scaling targetor ratio of a first DNN. Accordingly, the AI data includes informationfor verifying a down-scaling target of the first DNN.

Examples of the information included in the AI data include differenceinformation between resolution of the original image 105 and resolutionof the first image 115, and information related to the first image 115.

The difference information may be expressed as information about aresolution conversion degree of the first image 115 compared to theoriginal image 105 (for example, resolution conversion rateinformation). Also, because the resolution of the first image 115 isverified through the resolution of the reconstructed second image 135and the resolution conversion degree is verified accordingly, thedifference information may be expressed only as resolution informationof the original image 105. Here, the resolution information may beexpressed as vertical/horizontal sizes or as a ratio (16:9, 4:3, or thelike) and a size of one axis. Also, when there is pre-set resolutioninformation, the resolution information may be expressed in a form of anindex or flag through one or more signaled bits.

The information related to the first image 115 may include informationabout at least one of a bitrate of the image data obtained as the resultof performing first encoding on the first image 115 or a codec type usedduring the first encoding of the first image 115.

The AI up-scaler 234 may determine the up-scaling target of the secondimage 135, based on at least one of the difference information or theinformation related to the first image 115, which is included in the AIdata. The up-scaling target may indicate, for example, to what degreeresolution is to be up-scaled for the second image 135. When theup-scaling target is determined, the AI up-scaler 234 performs AIup-scaling on the second image 135 through a second DNN to obtain thethird image 145 corresponding to the up-scaling target.

Before describing a method, performed by the AI up-scaler 234, ofperforming AI up-scaling on the second image 135 according to theup-scaling target, an AI up-scaling process through the second DNN willbe described with reference to FIGS. 3 and 4.

FIG. 3 is a diagram showing a DNN 300 for performing AI up-scaling onthe second image 135, and FIG. 4 is a diagram for describing aconvolution operation in a first convolution layer 310 of FIG. 3.

As shown in FIG. 3, the second image 135 is input to the firstconvolution layer 310. The label 3X3X4 indicated in the firstconvolution layer 310 shown in FIG. 3 indicates that a convolutionprocess is performed on one input image by using four filter kernelshaving a size of 3×3. Four feature maps are generated by the four filterkernels as a result of the convolution process. Each feature mapindicates inherent characteristics of the second image 135. For example,each feature map may represent a vertical direction characteristic, ahorizontal direction characteristic, or an edge characteristic, etc. ofthe second image 135.

A convolution operation in the first convolution layer 310 will bedescribed in detail with reference to FIG. 4.

One feature map 450 may be generated through multiplication and additionbetween parameters of a filter kernel 430 having a size of 3×3 used inthe first convolution layer 310 and corresponding pixel values in thesecond image 135. Because four filter kernels are used in the firstconvolution layer 310, four feature maps may be generated through theconvolution operation using the four filter kernels.

In the second image 135 in FIG. 4, I1 through I49 indicate pixels in thesecond image 135, and F1 through F9 indicated in the filter kernel 430indicate parameters of the filter kernel 430. Also, M1 through M9indicated in the feature map 450 indicate samples of the feature map450.

In FIG. 4, the second image 135 includes 49 pixels, but the number ofpixels is only a simplified example and when the second image 135 has aresolution of 4 K, the second image 135 may include, for example,3840×2160 pixels.

During a convolution operation process, pixel values of I1, I2, I3, I8,I9, I10, I15, I16, and I17 of the second image 135 and F1 through F9 ofthe filter kernels 430 are respectively multiplied, and a value of thecombination (for example, addition) of result values of themultiplication may be assigned as a value of M1 of the feature map 450.When a stride of the convolution operation is 2, pixel values of I3, I4,I5, I10, I11, I12, I17, I18, and I19 of the second image 135 and F1through F9 of the filter kernels 430 are respectively multiplied, andthe value of the combination of the result values of the multiplicationmay be assigned as a value of M2 of the feature map 450.

While the filter kernel 430 proceeds according to the stride to the lastpixel of the second image 135, the convolution operation is performedbetween the pixel values in the second image 135 and the parameters ofthe filter kernel 430, and thus the feature map 450 having a certainsize may be generated.

According to the present disclosure, values of parameters of a secondDNN, for example, values of parameters of a filter kernel used inconvolution layers of the second DNN (for example, F1 through F9 of thefilter kernel 430), may be optimized through joint training of a firstDNN and the second DNN. As described above, the AI up-scaler 234 maydetermine an up-scaling target corresponding to a down-scaling target ofthe first DNN based on AI data, and determine parameters correspondingto the determined up-scaling target as the parameters of the filterkernel used in the convolution layers of the second DNN.

Convolution layers included in the first DNN and the second DNN mayperform processes according to the convolution operation processdescribed with reference to FIG. 4, but the convolution operationprocess described with reference to FIG. 4 is only an example and theconvolution operation process is not limited thereto.

Referring back to FIG. 3, the feature maps output from the firstconvolution layer 310 may be input to a first activation layer 320.

The first activation layer 320 may assign a non-linear feature to eachfeature map. The first activation layer 320 may include a sigmoidfunction, a Tan h function, a rectified linear unit (ReLU) function, orthe like, but the first activation layer 320 is not limited thereto.

The first activation layer 320 assigning the non-linear featureindicates that at least one sample value of the feature map, which is anoutput of the first convolution layer 310, is changed. Here, the changeis performed by applying the non-linear feature.

The first activation layer 320 determines whether to transmit samplevalues of the feature maps output from the first convolution layer 310to the second convolution layer 330. For example, some of the samplevalues of the feature maps are activated by the first activation layer320 and transmitted to the second convolution layer 330, and some of thesample values are deactivated by the first activation layer 320 and nottransmitted to the second convolution layer 330. The intrinsiccharacteristics of the second image 135 represented by the feature mapsare emphasized by the first activation layer 320.

Feature maps 325 output from the first activation layer 320 are input tothe second convolution layer 330. One of the feature maps 325 shown inFIG. 3 is a result of processing the feature map 450 described withreference to FIG. 4 in the first activation layer 320.

The label 3X3X4 indicated in the second convolution layer 330 indicatesthat a convolution process is performed on the feature maps 325 by usingfour filter kernels having a size of 3×3. An output of the secondconvolution layer 330 is input to a second activation layer 340. Thesecond activation layer 340 may assign a non-linear feature to inputdata.

Feature maps 345 output from the second activation layer 340 are inputto a third convolution layer 350. The label 3X3X1 indicated in the thirdconvolution layer 350 shown in FIG. 3 indicates that a convolutionprocess is performed to generate one output image by using one filterkernel having a size of 3×3. The third convolution layer 350 is a layerfor outputting a final image and generates one output by using onefilter kernel. According to an embodiment of the disclosure, the thirdconvolution layer 350 may output the third image 145 as a result of aconvolution operation.

There may be a plurality of pieces of DNN setting information indicatingthe numbers of filter kernels of the first, second, and thirdconvolution layers 310, 330, and 350 of the second DNN 300, a parameterof filter kernels of the first, second, and third convolution layers310, 330, and 350 of the second DNN 300, and the like, as will bedescribed later, and the plurality of pieces of DNN setting informationshould be connected to a plurality of pieces of DNN setting informationof a first DNN. The association between the plurality of pieces of DNNsetting information of the second DNN and the plurality of pieces of DNNsetting information of the first DNN may be realized via joint trainingof the first DNN and the second DNN.

In FIG. 3, the second DNN 300 includes three convolution layers (thefirst, second, and third convolution layers 310, 330, and 350) and twoactivation layers (the first and second activation layers 320 and 340),but this is only an example, and the configurations and quantities ofconvolution layers and activation layers may vary according to anembodiment. Also, according to an embodiment, the second DNN 300 may beimplemented as a recurrent neural network (RNN). In this case, aconvolutional neural network (CNN) structure of the second DNN 300according to an embodiment of the disclosure is changed to an RNNstructure.

According to an embodiment, the AI up-scaler 234 may include at leastone arithmetic logic unit (ALU) for the convolution operation and theoperation of the activation layer described above. The ALU may beimplemented as a processor. For the convolution operation, the ALU mayinclude a multiplier that performs multiplication between sample valuesof the second image 135 or the feature map output from previous layerand sample values of the filter kernel, and an adder that adds resultvalues of the multiplication. Also, for the operation of the activationlayer, the ALU may include a multiplier that multiplies an input samplevalue by a weight used in a pre-determined sigmoid function, a Tan hfunction, or an ReLU function, and a comparator that compares amultiplication result and a certain value to determine whether totransmit the input sample value to a next layer.

Hereinafter, a method, performed by the AI up-scaler 234, of performingthe AI up-scaling on the second image 135 according to the up-scalingtarget will be described.

According to an embodiment, the AI up-scaler 234 may store a pluralityof pieces of DNN setting information configurable in a second DNN.

Here, the DNN setting information may include information about at leastone of the number of convolution layers included in the second DNN, thenumber of filter kernels for each convolution layer, or a parameter ofeach filter kernel. The plurality of pieces of DNN setting informationmay respectively correspond to various up-scaling targets, and thesecond DNN may operate based on DNN setting information corresponding toa certain up-scaling target. The second DNN may have differentstructures based on the DNN setting information. For example, the secondDNN may include three convolution layers based on any piece of DNNsetting information, and may include four convolution layers based onanother piece of DNN setting information.

According to an embodiment, the DNN setting information may only includea parameter of a filter kernel used in the second DNN. In this case, thestructure of the second DNN does not change, but only the parameter ofthe internal filter kernel may change based on the DNN settinginformation.

The AI up-scaler 234 may obtain the DNN setting information forperforming AI up-scaling on the second image 135, among the plurality ofpieces of DNN setting information. Each of the plurality of pieces ofDNN setting information may be information for obtaining the third image145 of pre-determined resolution and/or pre-determined quality, and istrained jointly with a first DNN.

For example, one piece of DNN setting information among the plurality ofpieces of DNN setting information may include information for obtainingthe third image 145 of resolution twice the resolution of the secondimage 135, for example, the third image 145 of 4 K (4096×2160)resolution twice the 2 K (2048×1080) resolution of the second image 135,and another piece of DNN setting information may include information forobtaining the third image 145 of resolution four times higher than theresolution of the second image 135, for example, the third image 145 of8 K (8192×4320) resolution four times higher than the 2 K (2048×1080)resolution of the second image 135.

Each of the plurality of pieces of DNN setting information is obtainedjointly with DNN setting information of the first DNN of an AI encodingapparatus 600 of FIG. 6, and the AI up-scaler 234 obtains one piece ofDNN setting information among the plurality of pieces of DNN settinginformation according to an enlargement ratio corresponding to areduction ratio of the DNN setting information of the first DNN. In thisregard, the AI up-scaler 234 may verify information of the first DNN. Inorder for the AI up-scaler 234 to verify the information of the firstDNN, the AI decoding apparatus 200 according to an embodiment receivesAI data including the information of the first DNN from the AI encodingapparatus 600.

In other words, the AI up-scaler 234 may verify information targeted byDNN setting information of the first DNN used to obtain the first image115 and obtain the DNN setting information of the second DNN trainedjointly with the DNN setting information of the first DNN, by usinginformation received from the AI encoding apparatus 600.

When DNN setting information for performing the AI up-scaling on thesecond image 135 is obtained from among the plurality of pieces of DNNsetting information, input data may be processed based on the second DNNoperating according to the obtained DNN setting information.

For example, when any one piece of DNN setting information is obtained,the number of filter kernels included in each of the first, second, andthird convolution layers 310, 330, and 350 of the second DNN 300 of FIG.3, and the parameters of the filter kernels are set to values includedin the obtained DNN setting information.

In particular, parameters of a filter kernel of 3×3 used in any oneconvolution layer of the second DNN of FIG. 4 are set to {1, 1, 1, 1, 1,1, 1, 1, 1}, and when DNN setting information is changed afterwards, theparameters are replaced by {2, 2, 2, 2, 2, 2, 2, 2, 2} that areparameters included in the changed DNN setting information.

The AI up-scaler 234 may obtain the DNN setting information for AIup-scaling from among the plurality of pieces of DNN settinginformation, based on information included in the AI data, and the AIdata used to obtain the DNN setting information will now be described.

According to an embodiment, the AI up-scaler 234 may obtain the DNNsetting information for AI up-scaling from among the plurality of piecesof DNN setting information, based on difference information included inthe AI data. For example, when the resolution (for example, 4 K(4096×2160)) of the original image 105 is twice the resolution (forexample, 2 K (2048×1080)) of the first image 115, based on thedifference information, the AI up-scaler 234 may obtain the DNN settinginformation for increasing the resolution of the second image 135 twotimes.

According to another embodiment, the AI up-scaler 234 may obtain the DNNsetting information for AI up-scaling the second image 135 from amongthe plurality of pieces of DNN setting information, based on informationrelated to the first image 115 included in the AI data. The AI up-scaler234 may pre-determine a mapping relationship between image-relatedinformation and DNN setting information, and obtain the DNN settinginformation mapped to the information related to the first image 115.

FIG. 5 is a table showing a mapping relationship between image-relatedinformation and DNN setting information.

According to FIG. 5, it the AI encoding and AI decoding processesaccording to embodiments of the disclosure may consider more than only achange of resolution. As shown in FIG. 5, DNN setting information may beselected considering resolution, such as standard definition (SD), highdefinition (HD), or full HD, a bitrate, such as 10 Mbps, 15 Mbps, or 20Mbps, and codec information, such as AV1, H.264, or HEVC, individuallyor collectively. For such consideration of the resolution, the bitrateand the codec information, training in consideration of each elementshould be jointly performed with encoding and decoding processes duringan AI training process (see FIG. 9).

Accordingly, when a plurality of pieces of DNN setting information areprovided based on image-related information including a codec type,resolution of an image, and the like, as shown in FIG. 5 according totraining, the DNN setting information for AI up-scaling the second image135 may be obtained based on the information related to the first image115 received during the AI decoding process.

In other words, the AI up-scaler 234 is capable of using DNN settinginformation according to image-related information by correlating theimage-related information at the left of a table of FIG. 5 and the DNNsetting information at the right of the table.

As shown in FIG. 5, when it is verified, from the information related tothe first image 115, that the resolution of the first image 115 is SD, abitrate of image data obtained as a result of performing first encodingon the first image 115 is 10 Mbps, and the first encoding is performedon the first image 115 via AV1 codec, the AI up-scaler 234 may use A DNNsetting information among the plurality of pieces of DNN settinginformation.

Also, when it is verified, from the information related to the firstimage 115, that the resolution of the first image 115 is HD, the bitrateof the image data obtained as the result of performing the firstencoding is 15 Mbps, and the first encoding is performed via H.264codec, the AI up-scaler 234 may use B DNN setting information among theplurality of pieces of DNN setting information.

Also, when it is verified, from the information related to the firstimage 115, that the resolution of the first image 115 is full HD, thebitrate of the image data obtained as the result of performing the firstencoding is 20 Mbps, and the first encoding is performed via HEVC codec,the AI up-scaler 234 may use C DNN setting information among theplurality of pieces of DNN setting information. Last, when it isverified that the resolution of the first image 115 is full HD, thebitrate of the image data obtained as the result of performing the firstencoding is 15 Mbps, and the first encoding is performed via HEVC codec,the AI up-scaler 234 may use D DNN setting information among theplurality of pieces of DNN setting information. One of the C DNN settinginformation and the D DNN setting information may be selected based onwhether the bitrate of the image data obtained as the result ofperforming the first encoding on the first image 115 is 20 Mbps or 15Mbps. The different bitrates of the image data, obtained when the firstencoding is performed on the first image 115 of the same resolution viathe same codec, indicates different qualities of reconstructed images.Accordingly, a first DNN and a second DNN may be jointly trained basedon certain image quality, and accordingly, the AI up-scaler 234 mayobtain DNN setting information according to a bitrate of image dataindicating the quality of the second image 135.

According to another embodiment, the AI up-scaler 234 may obtain the DNNsetting information for performing AI up-scaling on the second image 135from among the plurality of pieces of DNN setting informationconsidering both information (prediction mode information, motioninformation, quantization parameter information, and the like) providedfrom the first decoder 232 and the information related to the firstimage 115 included in the AI data. For example, the AI up-scaler 234 mayreceive quantization parameter information used during a first encodingprocess of the first image 115 from the first decoder 232, verify abitrate of image data obtained as an encoding result of the first image115 from AI data, and obtain DNN setting information corresponding tothe quantization parameter information and the bitrate. Even when thebitrates are the same, the quality of reconstructed images may varyaccording to the complexity of an image. A bitrate is a valuerepresenting the entire first image 115 on which first encoding isperformed, and the quality of each frame may vary even within the firstimage 115. Accordingly, DNN setting information more suitable for thesecond image 135 may be obtained when prediction mode information,motion information, and/or a quantization parameter obtainable for eachframe from the first decoder 232 are considered together, compared towhen only the AI data is used.

Also, according to an embodiment, the AI data may include an identifierof mutually agreed DNN setting information. An identifier of DNN settinginformation is information for distinguishing a pair of pieces of DNNsetting information jointly trained between the first DNN and the secondDNN, such that AI up-scaling is performed on the second image 135 to theup-scaling target corresponding to the down-scaling target of the firstDNN. The AI up-scaler 234 may perform AI up-scaling on the second image135 by using the DNN setting information corresponding to the identifierof the DNN setting information, after obtaining the identifier of theDNN setting information included in the AI data. For example,identifiers indicating each of the plurality of DNN setting informationconfigurable or selectable in the first DNN and identifiers indicatingeach of the plurality of DNN setting information configurable orselectable in the second DNN may be previously designated. In this case,the same identifier may be designated for a pair of DNN settinginformation configurable or selectable in each of the first DNN and thesecond DNN. The AI data may include an identifier of DNN settinginformation set in the first DNN for AI down-scaling of the originalimage 105. The AI up-scaler 234 that receives the AI data may perform AIup-scaling on the second image 135 by using the DNN setting informationindicated by the identifier included in the AI data among the pluralityof DNN setting information.

Also, according to an embodiment, the AI data may include the DNNsetting information. The AI up-scaler 234 may perform AI up-scaling onthe second image 135 by using the DNN setting information afterobtaining the DNN setting information included in the AI data.

According to an embodiment, when pieces of information (for example, thenumber of convolution layers, the number of filter kernels for eachconvolution layer, a parameter of each filter kernel, and the like)constituting the DNN setting information are stored in a form of alookup table, the AI up-scaler 234 may obtain the DNN settinginformation by combining some values selected from values in the lookuptable, based on information included in the AI data, and perform AIup-scaling on the second image 135 by using the obtained DNN settinginformation.

According to an embodiment, when a structure of DNN corresponding to theup-scaling target is determined, the AI up-scaler 234 may obtain the DNNsetting information, for example, parameters of a filter kernel,corresponding to the structure of the DNN.

The AI up-scaler 234 obtains the DNN setting information of the secondDNN through the AI data including information related to the first DNN,and performs AI up-scaling on the second image 135 through the secondDNN set based on the obtained DNN setting information. As a result,memory usage and required throughput may be reduced as compared to whenfeatures of the second image 135 are directly analyzed for up-scaling.

According to an embodiment, when the second image 135 includes aplurality of frames, the AI up-scaler 234 may independently obtain DNNsetting information for a certain number of frames, or may obtain commonDNN setting information for entire frames.

FIG. 6 is a diagram showing an image including a plurality of frames.

As shown in FIG. 6, the second image 135 may include frames t0 throughtn.

According to an embodiment, the AI up-scaler 234 may obtain DNN settinginformation of a second DNN through AI data, and perform AI up-scalingon the frames t0 through tn based on the DNN setting information. Inother words, the frames t0 through tn may be processed via AI up-scalingbased on common DNN setting information.

According to another embodiment, the AI up-scaler 234 may perform AIup-scaling on some of the frames t0 through tn, for example, the framest0 through ta, by using ‘A’ DNN setting information obtained from AIdata, and perform AI up-scaling on the frames ta+1 through tb by using‘B’ DNN setting information obtained from the AI data. Also, the AIup-scaler 234 may perform AI up-scaling on the frames tb+1 through tn byusing ‘C’ DNN setting information obtained from the AI data. In otherwords, the AI up-scaler 234 may independently obtain DNN settinginformation for each group of frames including a certain number offrames among the plurality of frames, and perform AI up-scaling onframes included in each group by using the independently obtained DNNsetting information.

According to another embodiment, the AI up-scaler 234 may independentlyobtain DNN setting information for each frame forming the second image135. In other words, when the second image 135 includes three frames,the AI up-scaler 234 may perform AI up-scaling on a first frame by usingDNN setting information obtained in relation to the first frame, performAI up-scaling on a second frame by using DNN setting informationobtained in relation to the second frame, and perform AI up-scaling on athird frame by using DNN setting information obtained in relation to thethird frame. DNN setting information may be independently obtained foreach frame included in the second image 135, according to a method ofobtaining DNN setting information based on information (prediction modeinformation, motion information, quantization parameter information, orthe like) provided from the first decoder 232 and information related tothe first image 115 included in the AI data described above. This isbecause the mode information, the quantization parameter information, orthe like may be determined independently for each frame included in thesecond image 135.

According to another embodiment, the AI data may include informationabout to which frame DNN setting information obtained based on the AIdata is applicable. For example, when the AI data includes informationindicating that DNN setting information is valid up to the frame ta, theAI up-scaler 234 performs AI up-scaling on the frames t0 through ta byusing DNN setting information obtained based on the AI data. Also, whenanother piece of AI data includes information indicating that DNNsetting information is valid up to the frame tn, the AI up-scaler 234performs AI up-scaling on the frames ta+1 through tn by using DNNsetting information obtained based on the other piece of AI data.

Hereinafter, the AI encoding apparatus 600 for performing AI encoding onthe original image 105 will be described with reference to FIG. 7.

FIG. 7 is a block diagram of the AI encoding apparatus according to anembodiment.

Referring to FIG. 7, the AI encoding apparatus 600 may include an AIencoder 610 and a transmitter 630. The AI encoder 610 may include an AIdown-scaler 612 and a first encoder 614. The transmitter 630 may includea data processor 632 and a communicator 634.

In FIG. 7, the AI encoder 610 and the transmitter 630 are illustrated asseparate components or devices, but the AI encoder 610 and thetransmitter 630 may be jointly implemented through one processor forcontrolling the joint functionality of the AI encoder 610 and thetransmitter 630. In this case, the AI encoder 610 and the transmitter630 may be implemented through a dedicated processor or through acombination of software and a general-purpose processor such as AP, CPUor graphics processor GPU. The dedicated processor may be implemented byincluding a memory for implementing an embodiment of the disclosure orby including a memory processor for using an external memory.

Also, the AI encoder 610 and the transmitter 630 may be configured by aplurality of processors. In this case, the AI encoder 610 and thetransmitter 630 may be implemented through a combination of dedicatedprocessors or through a combination of software and a plurality ofgeneral-purpose processors such as AP, CPU or GPU. The AI down-scaler612 and the first encoder 614 may be implemented through differentprocessors.

The AI encoder 610 performs AI down-scaling on the original image 105and first encoding on the first image 115, and transmits AI data andimage data to the transmitter 630. The transmitter 630 transmits the AIdata and the image data to the AI decoding apparatus 200. Thetransmitter 630 may include a communication interface, such as awireless communication interface including a radio and/or an antenna ora wired communication interface such as HDMI, Ethernet, and the like.Alternatively, the transmitter 630 may include a portion or component ofa dedicated processor or through a combination of software and ageneral-purpose processor such as application processor (AP), centralprocessor (CPU) or graphic processor (GPU).

The image data includes data obtained as a result of performing thefirst encoding on the first image 115. The image data may include dataobtained based on pixel values in the first image 115, for example,residual data that is a difference between the first image 115 andprediction data of the first image 115. Also, the image data includesinformation used during a first encoding process of the first image 115.For example, the image data may include prediction mode information,motion information, quantization parameter information used to performthe first encoding on the first image 115, and the like.

The AI data includes information enabling AI up-scaling to be performedon the second image 135 to an up-scaling target corresponding to adown-scaling target of a first DNN. According to an embodiment, the AIdata may include difference information between the original image 105and the first image 115. Also, the AI data may include informationrelated to the first image 115. The information related to the firstimage 115 may include information about at least one of resolution ofthe first image 115, a bitrate of the image data obtained as the resultof performing the first encoding on the first image 115, or a codec typeused during the first encoding of the first image 115.

According to an embodiment, the AI data may include an identifier ofmutually agreed DNN setting information such that the AI up-scaling isperformed on the second image 135 to the up-scaling target correspondingto the down-scaling target of the first DNN.

Also, according to an embodiment, the AI data may include DNN settinginformation configurable in a second DNN.

The AI down-scaler 612 may obtain the first image 115 obtained byperforming the AI down-scaling on the original image 105 through thefirst DNN. The AI down-scaler 612 may determine the down-scaling targetof the original image 105, based on a pre-determined standard.

In order to obtain the first image 115 matching the down-scaling target,the AI down-scaler 612 may store a plurality of pieces of DNN settinginformation settable in the first DNN. The AI down-scaler 612 obtainsDNN setting information corresponding to the down-scaling target fromamong the plurality of pieces of DNN setting information, and performsthe AI down-scaling on the original image 105 through the first DNN setin the DNN setting information.

Each of the plurality of pieces of DNN setting information may betrained to obtain the first image 115 of pre-determined resolutionand/or pre-determined quality. For example, any one piece of DNN settinginformation among the plurality of pieces of DNN setting information mayinclude information for obtaining the first image 115 of resolution thatis half the resolution of the original image 105, for example, the firstimage 115 of 2 K (2048×1080) resolution that is half 4 K (4096×2160)resolution of the original image 105, and another piece of DNN settinginformation may include information for obtaining the first image 115 ofresolution that is a quarter the resolution of the original image 105,for example, the first image 115 of 2 K (2048×1080) resolution that is aquarter 8 K (8192×4320) resolution of the original image 105.

According to an embodiment, when pieces of information (for example, thenumber of convolution layers, the number of filter kernels for eachconvolution layer, a parameter of each filter kernel, and the like)constituting the DNN setting information are stored in a form of alookup table, the AI down-scaler 612 may obtain the DNN settinginformation by combining some values selected from values in the lookuptable, based on the down-scaling target, and perform AI down-scaling onthe original image 105 by using the DNN setting information.

According to an embodiment, the AI down-scaler 612 may determine astructure of DNN corresponding to the down-scaling target, and obtainDNN setting information corresponding to the structure of DNN, forexample, to configure parameters of a filter kernel.

The plurality of pieces of DNN setting information for performing the AIdown-scaling on the original image 105 may have an optimized value asthe first DNN and the second DNN are jointly trained. Here, each pieceof DNN setting information includes at least one of the number ofconvolution layers included in the first DNN, the number of filterkernels for each convolution layer, or a parameter of each filterkernel.

The AI down-scaler 612 may set the first DNN with the DNN settinginformation obtained for performing the AI down-scaling on the originalimage 105 to obtain the first image 115 of certain resolution and/orcertain quality through the first DNN. When the DNN setting informationfor performing the AI down-scaling on the original image 105 is obtainedfrom the plurality of pieces of DNN setting information, each layer inthe first DNN may process input data based on information included inthe DNN setting information.

Hereinafter, a method, performed by the AI down-scaler 612, ofdetermining the down-scaling target will be described. The down-scalingtarget may indicate, for example, by how much resolution is decreasedfrom the original image 105 to obtain the first image 115.

According to an embodiment, the AI down-scaler 612 may determine thedown-scaling target based on at least one of a compression ratio (forexample, a resolution difference between the original image 105 and thefirst image 115, target bitrate, or the like), compression quality (forexample, type of bitrate), compression history information, or a type ofthe original image 105.

For example, the AI down-scaler 612 may determine the down-scalingtarget based on the compression ratio, the compression quality, or thelike, which is pre-set or input from a user.

As another example, the AI down-scaler 612 may determine thedown-scaling target by using the compression history information storedin the AI encoding apparatus 600. For example, according to thecompression history information usable by the AI encoding apparatus 600,encoding quality, a compression ratio, or the like preferred by the usermay be determined, and the down-scaling target may be determinedaccording to the encoding quality determined based on the compressionhistory information. For example, the resolution, quality, or the likeof the first image 115 may be determined according to the encodingquality that has been used most often according to the compressionhistory information.

As another example, the AI down-scaler 612 may determine thedown-scaling target based on the encoding quality that has been usedmore frequently than a certain threshold value (for example, averagequality of the encoding quality that has been used more frequently thanthe certain threshold value), according to the compression historyinformation.

As another example, the AI down-scaler 612 may determine thedown-scaling target, based on the resolution, type (for example, a fileformat), or the like of the original image 105.

According to an embodiment, when the original image 105 includes aplurality of frames, the AI down-scaler 612 may independently determinea down-scaling target for a certain number of frames, or may determine adown-scaling target for entire frames.

According to an embodiment, the AI down-scaler 612 may divide the framesincluded in the original image 105 into a certain number of groups orsubsets, and independently determine the down-scaling target for eachgroup. The same or different down-scaling targets may be determined foreach group. The number of frames included in the groups may be the sameor different according to the each group.

According to another embodiment, the AI down-scaler 612 mayindependently determine a down-scaling target for each frame included inthe original image 105. The same or different down-scaling targets maybe determined for each frame.

Hereinafter, an example of a structure of a first DNN 700 on which AIdown-scaling is based will be described.

FIG. 8 is a diagram showing the DNN for performing AI down-scaling onthe original image.

As shown in FIG. 8, the original image 105 is input to a firstconvolution layer 710 of the first DNN 700. The first convolution layer710 performs a convolution process on the original image 105 by using 32filter kernels having a size of 5×5. Accordingly, 32 feature mapsgenerated as a result of the convolution process are input to a firstactivation layer 720. The first activation layer 720 may assign anon-linear feature to the 32 feature maps.

The first activation layer 720 determines whether to transmit samplevalues of the feature maps output from the first convolution layer 710to the second convolution layer 730. For example, some of the samplevalues of the feature maps are activated by the first activation layer720 and transmitted to the second convolution layer 730, and some of thesample values are deactivated by the first activation layer 720 and nottransmitted to the second convolution layer 730. Information representedby the feature maps output from the first convolution layer 710 isemphasized by the first activation layer 720.

An output 725 of the first activation layer 720 is input to a secondconvolution layer 730. The second convolution layer 730 performs aconvolution process on input data by using 32 filter kernels having asize of 5×5. Therefore, 32 feature maps output as a result of theconvolution process are input to a second activation layer 740, and thesecond activation layer 740 may assign a non-linear feature to the 32feature maps.

An output 745 of the second activation layer 740 is input to a thirdconvolution layer 750. The third convolution layer 750 performs aconvolution process on input data by using one filter kernel having asize of 5×5. As a result of the convolution process, one image may beoutput from the third convolution layer 750. The third convolution layer750 generates one output by using the one filter kernel as a layer foroutputting a final image. According to an embodiment of the disclosure,the third convolution layer 750 may output the first image 115 as aresult of a convolution operation.

There may be a plurality of pieces of DNN setting information indicatingthe numbers of filter kernels of the first, second, and thirdconvolution layers 710, 730, and 750 of the first DNN 700, a parameterof each filter kernel of the first, second, and third convolution layers710, 730, and 750 of the first DNN 700, and the like, and the pluralityof pieces of DNN setting information may be connected to a plurality ofpieces of DNN setting information of a second DNN. The connectionbetween the plurality of pieces of DNN setting information of the firstDNN and the plurality of pieces of DNN setting information of the secondDNN may be realized via joint training of the first DNN and the secondDNN.

In FIG. 8, the first DNN 700 includes three convolution layers (thefirst, second, and third convolution layers 710, 730, and 750) and twoactivation layers (the first and second activation layers 720 and 740),but this is only an example configuration, and the quantities andconfigurations of convolution layers and activation layers may varyaccording to an embodiment. Also, according to an embodiment, the firstDNN 700 may be implemented as an RNN. In this case, a CNN structure ofthe first DNN 700 according to an embodiment of the disclosure ischanged to an RNN structure.

According to an embodiment, the AI down-scaler 612 may include at leastone ALU for the convolution operation and the operation of theactivation layer described above. The ALU may be implemented as aprocessor. For the convolution operation, the ALU may include amultiplier that performs multiplication between sample values of theoriginal image 105 or the feature map output from previous layer andsample values of the filter kernel, and an adder that adds result valuesof the multiplication. Also, for the operation of the activation layer,the ALU may include a multiplier that multiplies an input sample valueby a weight used in a pre-determined sigmoid function, a Tan h function,or an ReLU function, and a comparator that compares a multiplicationresult and a certain value to determine whether to transmit the inputsample value to a next layer.

Referring back to FIG. 7, upon receiving the first image 115 from the AIdown-scaler 612, the first encoder 614 may reduce an information amountor quantity and size of data of the first image 115 by performing firstencoding on the first image 115. The image data corresponding to thefirst image 115 may be obtained as a result of performing the firstencoding by the first encoder 614.

The data processor 632 processes at least one of the AI data or theimage data to be transmitted in a certain form. For example, when the AIdata and the image data are to be transmitted in a form of a bitstream,the data processor 632 may process the AI data to be expressed in a formof a bitstream, and transmit the image data and the AI data in a form ofone bitstream through the communicator 634. As another example, the dataprocessor 632 may process the AI data to be expressed in a form ofbitstream, and transmit each of a bitstream corresponding to the AI dataand a bitstream corresponding to the image data through the communicator634. As another example, the data processor 632 may process the AI datato be expressed in a form of a frame or packet, and transmit the imagedata in a form of a bitstream and the AI data in a form of a frame orpacket through the communicator 634.

The communicator 634 transmits AI encoding data obtained as a result ofperforming AI encoding, through a network. The AI encoding data obtainedas the result of performing AI encoding includes the image data and theAI data. The image data and the AI data may be transmitted through asame type of network or different types of networks.

According to an embodiment, the AI encoding data obtained as a result ofprocesses of the data processor 632 may be stored in a data storagemedium including a magnetic medium such as a hard disk, a floppy disk,or a magnetic tape, an optical recording medium such as CD-ROM or DVD,or a magneto-optical medium such as a floptical disk.

Hereinafter, a method of jointly training the first DNN 700 and thesecond DNN 300 will be described with reference to FIG. 9.

FIG. 9 is a diagram for describing a method of training the first DNNand the second DNN.

In an embodiment, the original image 105 on which AI encoding isperformed through an AI encoding process is reconstructed to the thirdimage 145 via an AI decoding process, and in order to maintainconsistency between the original image 105 and the third image 145obtained as a result of AI decoding, coordination between the AIencoding process and the AI decoding process is required. In otherwords, information lost in the AI encoding process needs to bereconstructed during the AI decoding process, and in this regard, thefirst DNN 700 and the second DNN 300 need to be jointly trained tosimilarly account for the information loss.

For accurate AI decoding, ultimately, quality loss information 830corresponding to a result of comparing a third training image 804 and anoriginal training image 801 shown in FIG. 9 should be reduced.Accordingly, the quality loss information 830 is used to train both ofthe first DNN 700 and the second DNN 300.

First, a training process shown in FIG. 9 will be described.

In FIG. 9, the original training image 801 is an image on which AIdown-scaling is to be performed and a first training image 802 is animage obtained by performing AI down-scaling on the original trainingimage 801. Also, the third training image 804 is an image obtained byperforming AI up-scaling on the first training image 802.

The original training image 801 includes a still image or a moving imageincluding a plurality of frames. According to an embodiment, theoriginal training image 801 may include a luminance image extracted fromthe still image or the moving image including the plurality of frames.Also, according to an embodiment, the original training image 801 mayinclude a patch image extracted from the still image or the moving imageincluding the plurality of frames. When the original training image 801includes the plurality of frames, the first training image 802, thesecond training image, and the third training image 804 also eachinclude a plurality of frames. When the plurality of frames of theoriginal training image 801 are sequentially input to the first DNN 700,the plurality of frames of the first training image 802, the secondtraining image and the third training image 804 may be sequentiallyobtained through the first DNN 700 and the second DNN 300.

For joint training of the first DNN 700 and the second DNN 300, theoriginal training image 801 is input to the first DNN 700. The originaltraining image 801 input to the first DNN 700 is output as the firsttraining image 802 via the AI down-scaling, and the first training image802 is input to the second DNN 300. The third training image 804 isoutput as a result of performing the AI up-scaling on the first trainingimage 802.

Referring to FIG. 9, the first training image 802 is input to the secondDNN 300, and according to an embodiment, a second training imageobtained as first encoding and first decoding are performed on the firsttraining image 802 may be input to the second DNN 300. In order to inputthe second training image to the second DNN 300, any one codec amongMPEG-2, H.264, MPEG-4, H.265/HEVC, VC-1, VP8, VP9, and AV1 may be used.In particular, any one codec among MPEG-2, H.264, MPEG-4, H.265/HEVC,VC-1, VP8, VP9, and AV1 may be used to perform first encoding on thefirst training image 802 and first decoding on image data correspondingto the first training image 802.

Referring to FIG. 9, separate from the first training image 802 beingoutput through the first DNN 700, a reduced training image 803 obtainedby performing legacy down-scaling on the original training image 801 isobtained. Here, the legacy down-scaling may include at least one ofbilinear scaling, bicubic scaling, lanczos scaling, or stair stepscaling.

In order to prevent a structural feature of the first image 115 fromdeviating greatly from a structural feature of the original image 105,the reduced training image 803 is obtained to preserve the structuralfeature of the original training image 801.

Before training is performed, the first DNN 700 and the second DNN 300may be configured according to pre-determined DNN setting information.When the training is performed, structural loss information 810,complexity loss information 820, and the quality loss information 830may be determined.

The structural loss information 810 may be determined based on a resultof comparing the reduced training image 803 and the first training image802. For example, the structural loss information 810 may correspond toa difference between structural information of the reduced trainingimage 803 and structural information of the first training image 802.Structural information may include various features extractable from animage, such as luminance, contrast, histogram, or the like of the image.The structural loss information 810 indicates how much structuralinformation of the original training image 801 is maintained in thefirst training image 802. When the structural loss information 810 issmall, the structural information of the first training image 802 issimilar to the structural information of the original training image801.

The complexity loss information 820 may be determined based on spatialcomplexity of the first training image 802. For example, a totalvariance value of the first training image 802 may be used as thespatial complexity. The complexity loss information 820 is related to abitrate of image data obtained by performing first encoding on the firsttraining image 802. It is defined that the bitrate of the image data islow when the complexity loss information 820 is small.

The quality loss information 830 may be determined based on a result ofcomparing the original training image 801 and the third training image804. The quality loss information 830 may include at least one of anL1-norm value, an L2-norm value, a Structural Similarity (SSIM) value, aPeak Signal-To-Noise Ratio-Human Vision System (PSNR-HVS) value, aMultiscale SSIM (MS-SSIM) value, a Variance Inflation Factor (VIF)value, or a Video Multimethod Assessment Fusion (VMAF) value regardingthe difference between the original training image 801 and the thirdtraining image 804. The quality loss information 830 indicates howsimilar the third training image 804 is to the original training image801. The third training image 804 is more similar to the originaltraining image 801 when the quality loss information 830 is small.

Referring to FIG. 9, the structural loss information 810, the complexityloss information 820 and the quality loss information 830 are used totrain the first DNN 700, and the quality loss information 830 is used totrain the second DNN 300. In other words, the quality loss information830 is used to train both the first DNN 700 and the second DNN 300.

The first DNN 700 may update a parameter such that final lossinformation determined based on the structural loss information 810, thecomplexity loss information 820, and the quality loss information 830 isreduced or minimized. Also, the second DNN 300 may update a parametersuch that the quality loss information 830 is reduced or minimized.

The final loss information for training the first DNN 700 and the secondDNN 300 may be determined as Equation 1 below.

LossDS=a×Structural loss information+b×Complexity lossinformation+c×Quality loss information  [Equation 1]

LossUS=d×Quality loss information

In Equation 1, LossDS indicates final loss information to be reduced orminimized to train the first DNN 700, and LossUS indicates final lossinformation to be reduced or minimized to train the second DNN 300.Also, a, b, c and d may be pre-determined certain weights.

In other words, the first DNN 700 updates parameters such that LossDS ofEquation 1 is reduced, and the second DNN 300 updates parameters suchthat LossUS is reduced. When the parameters of the first DNN 700 areupdated according to LossDS derived during the training, the firsttraining image 802 obtained based on the updated parameters becomesdifferent from a previous first training image 802 obtained based onparameters that have not been updated, and accordingly, the thirdtraining image 804 also becomes different from a previous third trainingimage 804. When the third training image 804 becomes different from theprevious third training image 804, the quality loss information 830 isalso newly determined, and the second DNN 300 updates the parametersaccordingly. When the quality loss information 830 is newly determined,LossDS is also newly determined, and the first DNN 700 updates theparameters according to newly determined LossDS. In other words,updating of the parameters of the first DNN 700 leads to updating of theparameters of the second DNN 300, and updating of the parameters of thesecond DNN 300 leads to updating of the parameters of the first DNN 700.Consequently, because the first DNN 700 and the second DNN 300 arejointly trained by sharing the quality loss information 830, theparameters of the first DNN 700 and the parameters of the second DNN 300may be jointly optimized.

Referring to Equation 1, it is verified that LossUS is determinedaccording to the quality loss information 830, but this is only anexample and LossUS may be determined based on at least one of thestructural loss information 810 and the complexity loss information 820,and the quality loss information 830.

Hereinabove, it has been described that the AI up-scaler 234 of the AIdecoding apparatus 200 and the AI down-scaler 612 of the AI encodingapparatus 600 store the plurality of pieces of DNN setting information,and methods of training each of the plurality of pieces of DNN settinginformation stored in the AI up-scaler 234 and the AI down-scaler 612will now be described.

As described with reference to Equation 1, the first DNN 700 updates theparameters considering the similarity (the structural loss information810) between the structural information of the first training image 802and the structural information of the original training image 801, thebitrate (the complexity loss information 820) of the image data obtainedas a result of performing first encoding on the first training image802, and the difference (the quality loss information 830) between thethird training image 804 and the original training image 801.

In particular, the parameters of the first DNN 700 may be updated suchthat the first training image 802 having similar structural informationas the original training image 801 is obtained and the image data havinga small bitrate is obtained when first encoding is performed on thefirst training image 802, and at the same time, the second DNN 300performing AI up-scaling on the first training image 802 obtains thethird training image 804 similar to the original training image 801.

A direction in which values of the parameters of the first DNN 700 areoptimized may vary by adjusting the weights a, b, and c of Equation 1.For example, when the weight b is determined to be high, values of theparameters of the first DNN 700 may be updated by prioritizing a lowbitrate over high quality of the third training image 804. Also, whenthe weight c is determined to be high, values of the parameters of thefirst DNN 700 may be updated by prioritizing high quality of the thirdtraining image 804 over a high bitrate or maintaining of the structuralinformation of the original training image 801.

Also, the direction in which values of the parameters of the first DNN700 are optimized may vary according to a type of codec used to performfirst encoding on the first training image 802. This is because thesecond training image to be input to the second DNN 300 may varyaccording to the type of codec.

In other words, values of the parameters of the first DNN 700 and valuesof the parameters of the second DNN 300 may be jointly updated based onthe weights a, b, and c, and the type of codec for performing firstencoding on the first training image 802. Accordingly, when the firstDNN 700 and the second DNN 300 are trained after determining the weightsa, b, and c each to a certain value and determining the type of codec toa certain type, values of the parameters of the first DNN 700 andcorresponding values of the parameters of the second DNN 300 may bejointly optimized.

Also, when the first DNN 700 and the second DNN 300 are trained afterchanging the weights a, b, and c, and the type of codec, values theparameters of the first DNN 700 and values the parameters of the secondDNN 300 jointly optimized may be determined. In other words, theplurality of pieces of DNN setting information jointly trained with eachother may be determined in the first DNN 700 and the second DNN 300 whenthe first DNN 700 and the second DNN 300 are trained while changingvalues of the weights a, b, and c, and the type of codec.

As described above with reference to FIG. 5, the plurality of pieces ofDNN setting information of the first DNN 700 and the second DNN 300 maybe mapped to the information related to the first image. To set such amapping relationship, first encoding may be performed on the firsttraining image 802 output from the first DNN 700 via a certain codecaccording to a certain bitrate and the second training image obtained byperforming first decoding on a bitstream obtained as a result ofperforming the first encoding may be input to the second DNN 300. Inother words, by training the first DNN 700 and the second DNN 300 aftersetting an environment such that the first encoding is performed on thefirst training image 802 of a certain resolution via the certain codecaccording to the certain bitrate, a DNN setting information pair mappedto the resolution of the first training image 802, a type of the codecused to perform the first encoding on the first training image 802, andthe bitrate of the bitstream obtained as a result of performing thefirst encoding on the first training image 802 may be determined. Byvariously changing the resolution of the first training image 802, thetype of codec used to perform the first encoding on the first trainingimage 802 and the bitrate of the bitstream obtained according to thefirst encoding of the first training image 802, the mappingrelationships between the plurality of DNN setting information of thefirst DNN 700 and the second DNN 300 and the pieces of informationrelated to the first image may be determined.

FIG. 10 is a diagram for describing training processes of the first DNNand the second DNN by a training apparatus.

The training of the first DNN 700 and the second DNN 300 described withreference FIG. 9 may be performed by the training apparatus 1000. Thetraining apparatus 1000 may include the first DNN 700 and the second DNN300. The training apparatus 1000 may be, for example, the AI encodingapparatus 600 or a separate server. The DNN setting information of thesecond DNN 300 obtained as the training result may be stored in the AIdecoding apparatus 200.

Referring to FIG. 10, the training apparatus 1000 initially sets the DNNsetting information of the first DNN 700 and the second DNN 300, inoperations S840 and S845. Accordingly, the first DNN 700 and the secondDNN 300 may operate according to the initialized DNN settinginformation. The DNN setting information may include information aboutat least one of the number of convolution layers included in the firstDNN 700 and the second DNN 300, the number of filter kernels for eachconvolution layer, the size of a filter kernel for each convolutionlayer, or a parameter of each filter kernel.

The training apparatus 1000 provides the original training image 801 asinput to the first DNN 700, in operation S850. The original trainingimage 801 may include a still image or at least one frame included in amoving image.

The first DNN 700 processes the original training image 801 according tothe initialized DNN setting information and outputs the first trainingimage 802 obtained by performing AI down-scaling on the originaltraining image 801, in operation S855. In FIG. 10, the first trainingimage 802 output from the first DNN 700 is directly input to the secondDNN 300, but the first training image 802 output from the first DNN 700may be input to the second DNN 300 via the training apparatus 1000.Also, the training apparatus 1000 may perform first encoding and firstdecoding on the first training image 802 via a certain codec, andprovide the second training image as input to the second DNN 300.

The second DNN 300 processes the first training image 802 or the secondtraining image according to the initialized DNN setting information andoutputs the third training image 804 obtained by performing AIup-scaling on the first training image 802 or the second training image,in operation S860.

The training apparatus 1000 calculates the complexity loss information820, based on the first training image 802, in operation S865.

The training apparatus 1000 calculates the structural loss information810 by comparing the reduced training image 803 and the first trainingimage 802, in operation S870.

The training apparatus 1000 calculates the quality loss information 830by comparing the original training image 801 and the third trainingimage 804, in operation S875.

The initial DNN setting information is updated in operation S880 via afeedback propagation process based on the final loss information. Thetraining apparatus 1000 may calculate the final loss information fortraining the first DNN 700, based on the complexity loss information820, the structural loss information 810, and the quality lossinformation 830.

The second DNN 300 updates the DNN setting information in operation S885via a feedback propagation process based on the quality loss information830 or the final loss information. The training apparatus 1000 maycalculate the final loss information for training the second DNN 300,based on the quality loss information 830.

Then, the training apparatus 1000, the first DNN 700, and the second DNN300 may repeat operations S850 through S885 until the final lossinformation is minimized to update the DNN setting information. At thistime, during each repetition, the first DNN 700 and the second DNN 300operate according to the DNN setting information updated in the previousoperation.

Table 1 below shows effects when AI encoding and AI decoding areperformed on the original image 105 according to an embodiment of thedisclosure and when encoding and decoding are performed on the originalimage 105 via HEVC.

TABLE 1 Information Subjective Image Amount (Bitrate) Quality Score(Mbps) (VMAF) Frame AI Encoding/ AI Encoding/ Content Resolution NumberHEVC AI Decoding HEVC AI Decoding Content_01 8K 300 frames 46.3 21.494.80 93.54 Content_02 (7680 × 4320) 46.3 21.6 98.05 98.98 Content_0346.3 22.7 96.08 96.00 Content_04 46.1 22.1 86.26 92.00 Content_05 45.422.7 93.42 92.98 Content_06 46.3 23.0 95.99 95.61 Average 46.11 22.2594.10 94.85

As shown in Table 1, despite subjective image quality when AI encodingand AI decoding are performed on content including 300 frames of 8 Kresolution, according to an embodiment of the disclosure being higherthan subjective image quality when encoding and decoding are performedvia HEVC, a bitrate is still reduced by at least 50%.

FIG. 11 is a diagram of an apparatus for performing AI down-scaling onthe original image and an apparatus for performing AI up-scaling on thesecond image.

The apparatus 20 receives the original image 105 and provides image data25 and AI data 30 to the apparatus 40 by using an AI down-scaler 1124and a transformation-based encoder 1126. According to an embodiment, theimage data 25 corresponds to the image data of FIG. 1 and the AI data 30corresponds to the AI data of FIG. 1. Also, according to an embodiment,the transformation-based encoder 1126 corresponds to the first encoder614 of FIG. 7 and the AI down-scaler 1124 corresponds to the AIdown-scaler 612 of FIG. 7.

The apparatus 40 receives the AI data 30 and the image data 25 andobtains the third image 145 by using a transformation-based decoder 1146and an AI up-scaler 1144. According to an embodiment, thetransformation-based decoder 1146 corresponds to the first decoder 232of FIG. 2 and the AI up-scaler 1144 corresponds to the AI up-scaler 234of FIG. 2.

According to an embodiment, the apparatus 20 includes a CPU, a memory,and a computer program including instructions. The computer program isstored in the memory. According to an embodiment, the apparatus 20performs functions to be described with reference to FIG. 11 accordingto execution of the computer program by the CPU. According to anembodiment, the functions to be described with reference to FIG. 11 areperformed by a dedicated hardware chip and/or the CPU.

According to an embodiment, the apparatus 40 includes a CPU, a memory,and a computer program including instructions. The computer program isstored in the memory. According to an embodiment, the apparatus 40performs functions to be described with reference to FIG. 11 accordingto execution of the computer program by the CPU. According to anembodiment, the functions to be described with reference to FIG. 11 areperformed by a dedicated hardware chip and/or the CPU.

In FIG. 11, a configuration controller 1122 receives at least one inputvalue 10. According to an embodiment, the at least one input value 10may include at least one of a target resolution difference for the AIdown-scaler 1124 and the AI up-scaler 1144, a bitrate of the image data25, a bitrate type of the image data 25 (for example, a variable bitratetype, a constant bitrate type, or an average bitrate type), or a codectype for the transformation-based encoder 1126. The at least one inputvalue 10 may include a value pre-stored in the apparatus 20 or a valueinput from a user.

The configuration controller 1122 controls operations of the AIdown-scaler 1124 and the transformation-based encoder 1126, based on thereceived input value 10. According to an embodiment, the configurationcontroller 1122 obtains DNN setting information for the AI down-scaler1124 according to the received input value 10, and configures the AIdown-scaler 1124 with the DNN setting information. According to anembodiment, the configuration controller 1122 may transmit the inputvalue 10 to the AI down-scaler 1124 and the AI down-scaler 1124 mayobtain the DNN setting information for performing AI down-scaling on theoriginal image 105, based on the input value 10. According to anembodiment, the configuration controller 1122 may provide, to the AIdown-scaler 1124, additional information, for example, color format(luminance component, chrominance component, red component, greencomponent, or blue component) information to which AI down-scaling isapplied and tone mapping information of a high dynamic range (HDR),together with the input value 10, and the AI down-scaler 1124 may obtainthe DNN setting information considering the input value 10 and theadditional information. According to an embodiment, the configurationcontroller 1122 transmits at least a part of the received input value 10to the transformation-based encoder 1126 and the transformation-basedencoder 1126 performs first encoding on the first image 115 via abitrate of a certain value, a bitrate of a certain type, and a certaincodec.

The AI down-scaler 1124 receives the original image 105 and performs anoperation described with reference to at least one of FIG. 1, 7, 8, 9,or 10 to obtain the first image 115.

According to an embodiment, the AI data 30 is provided to the apparatus40. The AI data 30 may include at least one of resolution differenceinformation between the original image 105 and the first image 115, orinformation related to the first image 115. The resolution differenceinformation may be determined based on the target resolution differenceof the input value 10, and the information related to the first image115 may be determined based on at least one of a target bitrate, thebitrate type, or the codec type. According to an embodiment, the AI data30 may include parameters used during the AI up-scaling. The AI data 30may be provided from the AI down-scaler 1124 to the apparatus 40.

The image data 25 is obtained as the original image 105 is processed bythe transformation-based encoder 1126, and is transmitted to theapparatus 40. The transformation-based encoder 1126 may process thefirst image 115 according to MPEG-2, H.264 AVC, MPEG-4, H.265/HEVC,VC-1, VP8, VP9, or VA1.

A configuration controller 1142 controls an operation of the AIup-scaler 1144, based on the AI data 30. According to an embodiment, theconfiguration controller 1142 obtains the DNN setting information forthe AI up-scaler 1144 according to the received AI data 30, andconfigures the AI up-scaler 1144 according to the DNN settinginformation. According to an embodiment, the configuration controller1142 may transmit the received AI data 30 to the AI up-scaler 1144 andthe AI up-scaler 1144 may obtain the DNN setting information forperforming AI up-scaling on the second image 135, based on the AI data30. According to an embodiment, the configuration controller 1142 mayprovide, to the AI up-scaler 1144, additional information, for example,the color format (luminance component, chrominance component, redcomponent, green component, or blue component) information to which AIup-scaling is applied, and the tone mapping information of HDR, togetherwith the AI data 30, and the AI up-scaler 1144 may obtain the DNNsetting information considering the AI data 30 and the additionalinformation. According to an embodiment, the AI up-scaler 1144 mayreceive the AI data 30 from the configuration controller 1142, receiveat least one of prediction mode information, motion information, orquantization parameter information from the transformation-based decoder1146, and obtain the DNN setting information based on the AI data 30 andat least one of the prediction mode information, the motion information,and the quantization parameter information.

The transformation-based decoder 1146 may process the image data 25 toreconstruct the second image 135. The transformation-based decoder 1146may process the image data 25 according to MPEG-2, H.264 AVC, MPEG-4,H.265/HEVC, VC-1, VP8, VP9, or AV1.

The AI up-scaler 1144 may obtain the third image 145 by performing AIup-scaling on the second image 135 provided from thetransformation-based decoder 1146, based on the set DNN settinginformation.

The AI down-scaler 1124 may include a first DNN and the AI up-scaler1144 may include a second DNN, and according to an embodiment, DNNsetting information for the first DNN and second DNN are trainedaccording to the training method described with reference to FIGS. 9 and10.

An AI encoding technique and an AI decoding technique according to anembodiment of the disclosure have been described above with reference toFIGS. 1 through 11. Hereinafter, an AI encoding technique and an AIdecoding technique for performing up-scaling by updating up-scaling DNNsetting information optimized for an original image according to anotherembodiment of the disclosure will be described in detail with referenceto FIGS. 12 through 21. DNN setting information may include at least oneof the number of convolution layers, the number of filter kernels ineach of the convolution layers, a size of each filter kernel, orinformation about parameters of each filter kernel.

Hereinafter, a ‘parameter’ included in the ‘DNN setting information’ isa value used in a mathematical operation for each layer constituting aneural network, and may include, for example, a weight and a bias usedwhen an input value is applied to a predefined operation formula. Aweight may be a value for performing multiplication with an input value,and a bias may be a value for performing addition with a value obtainedas a result of performing multiplications between input values andweights. Also, parameters may be represented in a matrix form. Forexample, a weight may be a parameter of a 3×3 filter kernel used in aconvolution layer for AI up-scaling or AI down-scaling, and a bias maybe a parameter in a 1×1 matrix form that is added before an activationfunction is applied to a result obtained after performing a number ofconvolution operations equal to the number of input channels (depth) inthe convolution layer for AI up-scaling or AI down-scaling and addingresult values of each convolution operation together. Weights and biasesaccording to an embodiment of the disclosure will be described in moredetail below with reference to Equation 9. A parameter is a value set asa result of training and may be updated through separate training dataor training data composed of an original image.

Hereinafter, “model compression” refers to compression techniques forreducing the amount of data while maintaining the highest possibleaccuracy by reducing the number and size of parameters in an artificialneural network model, to reduce the complexity of the artificial neuralnetwork model. Examples of ‘model compression’ include pruning andquantization but model compression is not limited thereto.

FIG. 12 is a block diagram of a configuration of an AI encodingapparatus according to an embodiment of the disclosure. Referring toFIG. 12, the AI encoding apparatus 1200 includes an AI encoder 1210 anda transmitter 1230. The AI encoder 1210 includes a DNN settinginformation updater 1212, an AI down-scaler 1216, and a first encoder1214.

As described above, the AI down-scaler 1216 obtains a first image (e.g.,115 of FIG. 1) by performing AI down-scaling on an original image (e.g.,105 of FIG. 1) via a first DNN. The first image is an image having aresolution lower than that of the original image. Because AIdown-scaling by the AI down-scaler 1216 has been described above, aredundant description thereof is omitted below. The AI encodingapparatus 1200 may include a central processor for controlling the AIencoder 1210 and the transmitter 1230. Alternatively, the AI encoder1210 and the transmitter 1230 are operated by respective processors, andas the processors work closely together, the AI encoding apparatus 1200may be entirely operated. Alternatively, the AI encoder 1210 and thetransmitter 1230 may be controlled by an external processor.

The AI encoding apparatus 1200 may further include one or more datastorages or memories for storing data input to or output from the DNNsetting information updater 1212, the AI down-scaler 1216, the firstencoder 1214, and the transmitter 1230. The AI encoding apparatus 1200may further include a memory controller that controls input/output ofdata stored in a data storage.

To encode an image, the AI encoding apparatus 1200 may perform an imageencoding operation including prediction by interworking with a built-inor external video encoding processor. According to an embodiment of thedisclosure, the built-in video encoding processor of the AI encodingapparatus 1200 may implement a basic image encoding operation togetherwith a CPU or GPU including an image encoding processing module as wellas a separate processor.

The DNN setting information updater 1212 may update DNN settinginformation of a DNN for performing AI up-scaling corresponding to a DNNfor performing AI down-scaling. The DNN setting information updater 1212may update setting information for performing AI up-scaling of at leastone second image corresponding to at least one original image via jointtraining of an up-scaling DNN and a down-scaling DNN used for AIdown-scaling of the at least one original image or separate training ofthe up-scaling DNN by using the at least one original image as atraining image.

The DNN setting information updater 1212 may not select DNN settinginformation of a down-scaling DNN used for AI down-scaling of anoriginal image but determine the DNN setting information of thedown-scaling DNN and DNN setting information of an up-scaling DNN viajoint training of the up-scaling DNN and the down-scaling DNN by usingan original image as a training image.

In this case, the number of convolution layers, the number of filterkernels in each convolution layer, a size of each filter kernel, whichare included in DNN setting information, may be determined. Then, jointtraining may be performed based on the number of convolutional layers,the number of filter kernels in each of the convolutional layers, andthe size of each filter kernel. For example, the number of convolutionlayers, the number of filter kernels in each convolution layer, and asize of each filter kernel may be determined based on a structure of amost complex up-scaling DNN included in default up-scaling DNN settinginformation. In other words, the number of convolutional layers may bedetermined to be less than or equal to the number of convolutionallayers in the most complex up-scaling DNN, which is included in thedefault up-scaling DNN setting information, and the number of filterkernels in each convolution layer may be determined to be less than orequal to the number of filter kernels in each of the most complexconvolution layers included in the default up-scaling DNN settinginformation. The size of each filter kernel may be determined to be lessthan or equal to a size of each filter kernel in the most complexup-scaling DNN included in the default up-scaling DNN settinginformation.

Furthermore, in this case, initial parameters included in DNN settinginformation of a down-scaling/up-scaling DNN needed for joint trainingmay be randomly initialized parameters. For example, initial parametersof a down-scaling/up-scaling DNN may be values randomly sampled based ona particular probability distribution. For example, weights among theinitial parameters of the down-scaling/up-scaling DNN may be valuesrandomly sampled based on a Gaussian probability distribution.Furthermore, for example, a bias among the initial parameters of thedown-scaling/up-scaling DNN may be initialized to 0.

Alternatively, for example, an initial parameter of thedown-scaling/up-scaling DNN needed for joint training may be one of aplurality of predetermined down-scaling/up-scaling DNN parameters.Alternatively, an initial parameter of the down-scaling/up-scaling DNNneeded for joint training may be a default down-scaling/up-scaling DNNparameter. Alternatively, an initial parameter of thedown-scaling/up-scaling DNN needed for joint training may be a parameterof the down-scaling/up-scaling DNN, which is used in an immediatelypreceding group of pictures (GOP) unit, an immediately preceding intrarandom access point (IRAP) period unit, an immediately precedingsequence unit, an immediately preceding unit of a preset number offrames, or the like.

An image input to an up-scaling DNN during joint training may be thefirst image generated by down-scaling the original image via adown-scaling DNN, but the image is not limited thereto. The input imagemay be a second image (e.g., 135 of FIG. 1) generated by performingfirst encoding and first decoding on the first image obtained after thedown-scaling.

Thereafter, the DNN setting information updater 1212 may selectdetermined parameters of the down-scaling DNN and determine parametersof an up-scaling DNN via separate training of the up-scaling DNN byusing at least one original image as a training image. In this case,initial parameters of the up-scaling DNN available before the separatetraining may be parameters of the up-scaling DNN determined during jointtraining, but the initial parameters are not limited thereto. Weights inthe up-scaling DNN may be values randomly sampled based on a particularprobability distribution, and a bias in the up-scaling DNN may bedetermined to be 0.

An image input to the up-scaling DNN for separate training of theup-scaling DNN during the separate training may be the second imagegenerated by performing first encoding and first decoding on the firstimage generated by down-scaling the original image via a down-scalingDNN.

In this case, parameters of the up-scaling DNN, which are finallygenerated during or after training, may be generated via modelcompression.

Model compression used herein refers to compression techniques forreducing the amount of data while maintaining the highest possibleaccuracy, by reducing the number and size of parameters in an artificialneural network model, to reduce the complexity of the artificial neuralnetwork model. Information loss may occur during model compression.Representative examples of model compression include quantization andpruning, but the model compression is not limited thereto. Quantizationrefers to the process of dividing a continuous variation into a finitenumber of levels that change discontinuously and assigning a uniquevalue to each level. A typical example of quantization is the process ofconverting a non-integer value into an integer value via a roundingoperation such as rounding off, rounding down, and rounding up. Indetail, quantization may include transforming a data type of parameterinformation of the up-scaling DNN, and for example, may includetransforming a real data type such as float32 and double64 into aninteger data type such as int16 and int8.

Pruning refers to the process of changing a value having a small amountof data to zero. A typical example of pruning includes an operation ofsetting to 0 a DNN parameter value that is less than a certain valueclose to 0 and is included in the parameter information of theup-scaling DNN, but pruning is not limited thereto.

Alternatively, instead of performing joint training of a down-scalingDNN and an up-scaling DNN, the DNN setting information updater 1212 mayselect parameters of the down-scaling DNN and determine parameters ofthe up-scaling DNN for up-scaling at least one original image viaseparate training by using the at least one original image as a trainingimage. For example, an initial parameter of the down-scaling DNN neededfor separate training may be one of a plurality of predetermineddown-scaling DNN parameters or default down-scaling DNN parameters.Furthermore, as initial parameters of the up-scaling DNN availablebefore separate training, weights in the up-scaling DNN may be valuesrandomly sampled based on a particular probability distribution, and abias in the up-scaling DNN may be determined to be 0.

However, embodiments of the disclosure are not limited thereto, and aninitial parameter of the down-scaling/up-scaling DNN needed for separatetraining may be a parameter of the down-scaling/up-scaling DNN, which isused in an immediately preceding group of pictures (GOP) unit, animmediately preceding instantaneous decoding refresh (IDR) pictureperiod (IRAP) period unit, an immediately preceding sequence unit, animmediately preceding unit of a preset number of frames, or the like.

In this case, an image input to the up-scaling DNN for separate trainingof the up-scaling DNN may be the second image generated by performingfirst encoding and first decoding on the first image generated bydown-scaling the original image via a down-scaling DNN.

The DNN setting information updater 1212 may generate a first trainingimage via the down-scaling DNN by using at least one original image as atraining image, generate a third training image via the up-scaling DNNby using the first training image, and update the up-scaling DNN and thedown-scaling DNN based on first loss information and third lossinformation, each corresponding to a result of comparing the thirdtraining image with the original image that has not undergone AIdown-scaling, and second loss information generated based on the firsttraining image. Here, the first loss information may be informationbased on a result of comparing a quality parameter of the third trainingimage with a quality parameter of the original image. In other words,when the first loss information has a smaller value, this may indicatethat a quality of the original image is more similar to a quality of thethird training image. The first loss information may be information forusing specialized structural information of the original image fortraining. The first loss information may correspond to the quality lossinformation 830 of FIG. 9.

In this case, the result of the comparing of the quality parameters maybe a quality comparison parameter such as an L1-norm value, an L2-normvalue, an SSIM value, a PSNR-HVS value, an MS-SSIM value, a VIF value,or a VMAF value regarding a difference between the third training imageand the original image, but the quality comparison parameter is notlimited thereto. The result of the comparing of the quality parametersmay be determined based on a combination of the quality comparisonparameters. For example, the first loss information may be determinedbased on Equation 2 below. In Equation 2, i, j, k each denote a weightfor adjusting relative importance of each quality comparison parameter.

FIRST LOSS INFORMATION=i*L1-norm+j*SSIM,i*L2-norm+j*VIF; ori*L1-norm+j*L2-norm+k*SSIM  [Equation 2]

The third loss information may be information based on a result ofcomparing a feature-related parameter of the third training image with afeature-related parameter of the original image. In other words, whenthe third loss information has a smaller value, this may indicate thatfeatures of the original image are more similar to features of the thirdtraining image. The third loss information may be information for usingspecialized detail information (i.e., feature-related information) ofthe original image for training.

For example, the result of comparing the feature-related parameters maybe an L1-norm value or an L2-norm value of features of each image, butthe result of comparing the feature-related parameters is not limitedthereto. Edge information determined based on a Sobel edge detector orCanny edge detector may be a feature-related parameter.

The second loss information may be information related to a spatialcomplexity of the first training image. For example, the second lossinformation may represent a total variance of the first training image.The smaller the second loss, the smaller the amount of image datatransmitted through a bitstream. The second loss information maycorrespond to the complexity loss information 820 of FIG. 9.

Thus, the first loss information and the third loss information areinformation related to distortion, while the second loss information isinformation related to a rate, and when training is performed byconsidering all of the first loss information, the second lossinformation, and the third loss information, the DNN setting informationupdater 1212 may determine an optimal up-scaling DNN parameter based ona rate-distortion (RD) cost.

The DNN setting information updater 1212 may determine a loss valueLoss_(DS) based on Equation 3 below and update, based on the loss valueLoss_(DS), parameters of the up-scaling DNN and the down-scaling DNNsuch that the loss value Loss_(DS) decreases.

Loss_(DS) =a*FIRST LOSS INFORMATION+b*SECOND LOSS INFORMATION+c*THIRDLOSS INFORMATION  [Equation 3]

Moreover, the DNN setting information updater 1212 may determine a lossvalue Loss_(DS) based on Equation 4 below by additionally consideringfourth loss information and update, based on the loss value Loss_(DS),parameters of the up-scaling DNN and the down-scaling DNN such that theloss value Loss_(DS) decreases. In this case, the fourth lossinformation may correspond to the structural loss information 810 ofFIG. 9.

Loss_(DS) =a*FIRST LOSS INFORMATION+b*SECOND LOSS INFORMATION+c*THIRDLOSS INFORMATION+d*FOURTH LOSS INFORMATION  [Equation 4]

The DNN setting information updater 1212 may select parameters of thedown-scaling DNN, perform first encoding on a first training imageobtained by down-scaling at least one original image via thedown-scaling DNN, generate a third training image by up-scaling, via theup-scaling DNN, a second training image obtained by performing firstdecoding on the first training image that has undergone the firstencoding, and update the up-scaling DNN based on first loss informationand third loss information, each corresponding to a result of comparingthe third training image with the original image that has not undergoneAI down-scaling. The first loss information and the third lossinformation may be the first loss information and third loss informationdescribed above.

For example, the DNN setting information updater 1212 may determine aloss value Loss_(US) based on Equation 5 below and update, based on theloss value Loss_(US), parameters of the up-scaling DNN such that theloss value Loss_(US) decreases.

Loss_(US) =d*FIRST LOSS INFORMATION+e*THIRD LOSS INFORMATION  [Equation5]

The DNN setting information updater 1212 may generate information aboutparameters of the up-scaling DNN obtained via joint training of thedown-scaling DNN and the up-scaling DNN or separate training of theup-scaling DNN.

Furthermore, the DNN setting information updater 1212 may generateinformation about at least one of the number of convolutional layers,the number of filter kernels in each convolutional layer, or a size ofeach filter kernel.

The DNN setting information updater 1212 may encode DNN settinginformation of the up-scaling DNN. For example, the DNN settinginformation updater 1212 may encode information about at least one ofthe number of convolution layers, the number of filter kernels in eachconvolution layer, or the size of each filter kernel. Furthermore, theDNN setting information updater 1212 may encode information aboutparameters of the up-scaling DNN. For example, the DNN settinginformation updater 1212 may encode a weight residual or a bias residualof a filter kernel in the up-scaling DNN. In this case, the weightresidual or bias residual may be a difference between a weight or a biasencoded in an immediately preceding period unit and a weight or a biasdetermined for a current period unit.

In this case, encoding may be performed using differential pulse codemodulation (DPCM), run-length coding (RLC), and Huffman coding, but theencoding is not limited thereto. The encoding may be performed usingvarious other entropy encoding techniques.

Embodiments of the disclosure are not limited thereto, and the DNNsetting information updater 1212 may perform entropy encoding on DNNsetting information of the up-scaling DNN based on a context model. Forexample, the DNN setting information updater 1212 may generate a contextmodel based on DNN setting information of the up-scaling DNN for apreceding period unit, update the DNN setting information of theup-scaling DNN for a current period unit, and entropy-encode the updatedDNN setting information of the up-scaling DNN by using the contextmodel. In this case, the context model refers to a model representingthe probability of occurrence of a symbol estimated based on variouspieces of context information of the surrounding environment.

The context model may be updated each time weights or a bias of a filterkernel in the up-scaling DNN for the current period unit are encoded,and information about next weights or biases of a filter kernel in theup-scaling DNN for the current period unit may be entropy-encoded basedon the updated context model. The context model may be determined forthe entire up-scaling DNN, but the determination is not limited thereto,and the context model may be determined for each convolutional layer,for at least one filter kernel corresponding to one output channel in aconvolutional layer, and for each filter kernel in the convolutionallayer. Examples of entropy encoding based on contexts may include rangecoding, arithmetic coding, and Huffman coding, but the entropy encodingis not limited thereto.

The DNN setting information updater 1212 may update weights or biases offilter kernels in all layers of the up-scaling DNN and encodeinformation about the updated weights or biases of the filter kernels inall the layers, but the updating is not limited thereto. The DNN settinginformation updater 1212 may update weights or biases in some of thelayers and encode information about the updated weights or biases in thecorresponding layers. Alternatively, the DNN setting information updater1212 may update weights or biases of some filter kernels in a layer andencode information about the updated weights or biases of some filterkernels. For example, the DNN setting information updater 1212 mayupdate weights or biases in a last layer of the up-scaling DNN andencode information about the updated weights or biases in the lastlayer. The DNN setting information updater 1212 may encode updated DNNsetting information of the up-scaling DNN, including weight residualinformation or bias residual information. The weight residualinformation or bias residual information indicates a difference betweena weight or a bias of all or some of the filter kernels in all or someof the convolution layers in the up-scaling DNN before the weight or thebias is updated, and a weight or a bias of all or some of the filterkernels in all or some of the convolution layers in the up-scaling DNNafter the weight or the bias is updated.

The DNN setting information updater 1212 may perform frequencytransformation on a weight residual and encode updated DNN settinginformation of the up-scaling DNN, including information about theweight residual that has undergone the frequency transformation.

The DNN setting information updater 1212 may perform entropy encoding ona weight or a bias updated for use in the current period unit based on acontext model, but the entropy encoding is not limited thereto. The DNNsetting information updater 1212 may entropy-encode, based on thecontext model, a difference between a weight or a bias updated for usein the current period unit and a weight or a bias used in theimmediately preceding period unit. In this case, the context model maybe based on information about a difference between weights or biasesdetermined for use in the up-scaling DNN in at least one of theimmediately preceding period unit or a period unit preceding theimmediately preceding period unit.

The DNN setting information updater 1212 may entropy-encode weightinformation or bias information in combination with a model compressiontechnique (e.g., pruning or quantization).

The DNN setting information updater 1212 may periodically update DNNsetting information of the up-scaling DNN. For example, the DNN settinginformation updater 1212 may update DNN setting information for eachGOP, each IRAP period (e.g., instantaneous decoding refresh (IDR)picture period), each sequence, or each unit of a preset number offrames.

For example, the AI encoding apparatus 1200 may perform scene changedetection while encoding an image every IDR picture period, and when ascene change is detected in a frame, determine the frame as an IDRpicture and encode subsequent images every IDR picture period. In thiscase, the DNN setting information updater 1212 may update DNN settinginformation for each IDR picture period in which the scene change isdetected.

The DNN setting information updater 1212 may update the DNN settinginformation whenever necessary. For example, the DNN setting informationupdater 1212 may determine whether to update the DNN setting informationfor each preset period and update the DNN setting information when theDNN setting information updater 1212 determines to update the DNNsetting information. The DNN setting information updater 1212 maydetermine DNN setting information of the up-scaling DNN based on atleast some of at least one original image and determine whether toupdate DNN setting information by comparing, based on an RD cost, aresult of performing AI down-scaling and AI up-scaling based on thedetermined DNN setting information with a result of performing AIdown-scaling and AI up-scaling based on DNN setting information that hasnot been updated.

In this case, the DNN setting information updater 1212 may encode flaginformation regarding whether to update the DNN setting information.When the DNN setting information updater 1212 determines to update theDNN setting information, the DNN setting information updater 1212 mayencode the flag information as a first value. On the other hand, whenthe DNN setting information updater 1212 determines not to update theDNN setting information, the DNN setting information updater 1212 mayencode the flag information as a second value.

When the DNN setting information updater 1212 determines not to updatethe DNN setting information, the flag information may not be encoded.The first or second value may be 0 or 1, but the value is not limitedthereto.

The DNN setting information updater 1212 may determine whether thesecond image is to be up-scaled by using predetermined DNN settinginformation of an AI up-scaling DNN or by using DNN setting informationof the AI up-scaling DNN, which is updated using an original image as atraining image. Herein, the terms “AI up-scaling DNN” and “up-scalingDNN” or “AI down-scaling DNN” and “down-scaling DNN” may be usedinterchangeably. The DNN setting information updater 1212 may determineDNN setting information based on at least some of at least one originalimage and determine whether the second image is to be up-scaled based onthe predetermined DNN setting information of the AI up-scaling DNN orbased on the DNN setting information of the AI up-scaling DNN, which isupdated using an original image as a training image, by comparing, basedon an RD cost, a result of performing AI down-scaling and AI up-scalingbased on the determined DNN setting information with a result ofperforming AI down-scaling and AI up-scaling based on the predeterminedDNN setting information.

The DNN setting information updater 1212 may encode flag informationindicating whether the second image is to be up-scaled based on thepredetermined DNN setting information of the AI up-scaling DNN or basedon the DNN setting information of the AI up-scaling DNN, which isupdated using the original image as a training image. When the flaginformation has a first value, the flag information may indicate thatthe second image is to be up-scaled by using the predetermined settinginformation of the AI up-scaling DNN, and when the flag information hasa second value, the flag information may indicate that the second imageis to be up-scaled by using the DNN setting information of the AIup-scaling DNN, which is updated using the original image as a trainingimage. When the second image is to be up-scaled by using thepredetermined setting information of the AI up-scaling DNN, the DNNsetting information updater 1212 may not encode the flag information.

The DNN setting information updater 1212 may update DNN settinginformation of the AI up-scaling DNN based on at least one of aresolution of the original image or the first image or a bitrate of theimage data.

For example, the DNN setting information updater 1212 may change a DNNstructure by varying at least one of the number of convolutional layers,the number of filter kernels in each convolution layer, or a size ofeach filter kernel based on at least one of a resolution of the originalimage or the first image or a bitrate of the image data. The AI encodingapparatus 1200 may update DNN setting parameters by performing trainingof the up-scaling DNN based on the changed DNN structure. For example, aDNN structure may be changed to have 3 convolution layers, filterkernels of a 3×3 size, and 4 channels, and DNN setting informationincluding setting parameters of the AI up-scaling DNN may be updatedbased on the configured DNN structure. The DNN structure may also bechanged to include 4 convolution layers, filter kernels of a 3×3 size,and 8 channels, and the setting parameters of the AI up-scaling DNN maybe updated based on the changed DNN structure. Furthermore, the DNNstructure may be configured to include 5 convolution layers, filterkernels of a 5×5 size, and 8 channels, and the setting parameters of theAI up-scaling DNN may be updated based on the changed DNN structure.When joint training of the up-scaling DNN and the down-scaling DNN isperformed by taking into account the same bitrate of image data and thesame resolutions of the original image and the first image, the imagedata to be encoded may vary accordingly as the DNN structure changes.However, in the case of separate training of the up-scaling DNN, duringwhich the first image that has undergone down-scaling remains selectedeven when the DNN structure changes, the image data may remain intact.Furthermore, in the case of separate training of the up-scaling DNN, asthe DNN setting information is updated, the quality of the third image(145 of FIG. 1), which is an image obtained by an AI decoding apparatus(1300 of FIG. 13) after performing AI up-scaling, may be furtherimproved.

Moreover, the DNN setting information updater 1212 may generate piecesof DNN setting information for a plurality of DNN structures by changinga DNN structure in various ways and may encode the pieces of DNN settinginformation for the plurality of DNN structures. In this case, theplurality of DNN structures may be DNN structures corresponding to thesame bitrate of image data and the same resolutions of the originalimage and the first image. However, embodiments of the disclosure arenot limited thereto, and the plurality of DNN structures may be DNNstructures corresponding to various bitrates of image data and variousresolutions of the original image and the first image.

Although it has been described with reference to FIG. 12 that the DNNsetting information updater 1212 encodes the DNN setting information ofthe up-scaling DNN, embodiments of the disclosure are not limitedthereto, and it will be understood by those of ordinary skill in the artthat the DNN setting information updater 1212 may transmit the DNNsetting information of the up-scaling DNN to a data processor 1232 andthe data processor 1232 may perform the above-described encodingoperation of the DNN setting information updater 1212.

The first encoder 1214 encodes the first image obtained by the AIdown-scaler 1216 performing down-scaling.

As described above, the encoding may include a process of generatingprediction data by predicting the first image, a process of generatingresidual data corresponding to a difference between the first image andthe prediction data, a process of transforming the residual data in aspatial domain into a frequency domain component, a process ofquantizing the residual data that has undergone the transformation intothe frequency domain component, and a process of entropy-encoding thequantized residual data.

The data processor 1232 processes at least one of AI data or image datato be transmitted in a predefined format. For example, when the AI dataand the image data need to be transmitted in the form of a bitstream,the data processor 1232 may process the AI data so that the AI data isrepresented in the form of a bitstream and transmit the AI data and theimage data in the form of one bitstream via the communicator 1234. Asanother example, the data processor 1232 may process the AI data so thatthe AI data is represented in the form of a bitstream and respectivelytransmit a bitstream corresponding to the AI data and a bitstreamcorresponding to the image data via the communicator 1234. As anotherexample, the data processor 1232 may process the AI data so that the AIdata is represented as a frame or packet and transmit the image data inthe form of a bitstream and AI data in the form of a frame or packet viathe communicator 1234.

The communicator 1234 transmits AI encoding data generated as a resultof the AI encoding via a network. The AI encoding data generated as aresult of the AI encoding includes the image data and the AI data.

The image data and the AI data may be transmitted via a homogeneous orheterogeneous network.

The image data includes data generated as a result of the first encodingof the first image. The image data may include data generated based onpixel values in the first image, such as residual data that is adifference between the first image and the prediction data. Furthermore,the image data includes information used during the first encoding ofthe first image. For example, the image data may include modeinformation and information related to quantization parameters, whichare used to perform the first encoding on the first image.

The AI data includes pieces of information that enable the AI decodingapparatus 1300 to perform AI up-scaling on the second image according toan up-scaling target corresponding to a down-scaling target of a firstDNN (a down-scaling DNN). For example, the AI data may includedifference information between the original image and the first image.Furthermore, the AI data may include information related to the firstimage. The information related to the first image may includeinformation about at least one of a resolution of the first image, abitrate of image data generated as a result of the first encoding of thefirst image, or a codec type used in the first encoding of the firstimage.

FIG. 13 is a block diagram of a configuration of the AI decodingapparatus according to an embodiment of the disclosure. Referring toFIG. 13, the AI decoding apparatus 1300 includes a receiver 1310 and anAI decoder 1330. The AI decoder 1330 includes a first decoder 1332, anAI up-scaler 1334, and a DNN setting information updater 1336.

The AI decoding apparatus 1300 may include a central processor thatcontrols the receiver 1310 and the AI decoder 1330. Alternatively, thereceiver 1310 and the AI decoder 1330 are operated by respectiveprocessors, and as the processors work closely together, the AI decodingapparatus 1300 may be completely operated. Alternatively, the receiver1310 and the AI decoder 1330 may be controlled by an external processor.

The AI decoding apparatus 1300 may further include one or more datastorages for storing data input to or output from the DNN settinginformation updater 1336, the AI up-scaler 1334, the first decoder 1332,and the receiver 1310. The AI decoding apparatus 1300 may furtherinclude a memory controller that controls input/output of data stored ina data storage.

To decode an image, the AI decoding apparatus 1300 may perform an imagedecoding operation including prediction by interworking with a built-inor external video decoding processor. According to an embodiment of thedisclosure, the built-in video decoding processor of the AI decodingapparatus 1300 may implement a basic image decoding operation togetherwith a CPU or GPU including an image decoding processing module as wellas a separate processor.

The communicator 1312 receives AI encoding data including image data andAI data via a network. The image data includes information generated asa result of the first encoding of the first image, and the AI dataincludes DNN setting information of the up-scaling DNN.

The parser 1314 divides the AI-encoding data received via thecommunicator 1312 into the image data and the AI data and transmits theimage data to the first decoder 1332 and the AI data to the AI up-scaler1334 via the outputter 1316.

Because operations of the communicator 1312, the parser 1314, and theoutputter 1316 of the AI decoding apparatus 1300 of FIG. 13 respectivelycorrespond to those of the communicator 212, the parser 214, and theoutputter 216 of the AI decoding apparatus 200 described with referenceto FIG. 2, redundant descriptions are omitted.

The AI up-scaler 1334 obtains a third image (e.g., 145 of FIG. 1) byperforming AI up-scaling on a second image via a second DNN (anup-scaling DNN). The third image is an image having a higher resolutionthan that of the second image. Because the AI up-scaling by the AIup-scaler 1334 has been described above, a redundant description thereofis omitted.

The DNN setting information updater 1336 may be provided separately fromthe AI up-scaler 1334, but the DNN setting information updater 1336 notlimited thereto, and the DNN setting information updater 1336 may beincluded in the AI up-scaler 1334.

The DNN setting information updater 1336 may update, based on DNNsetting information, DNN setting information of the AI up-scaling DNNcorresponding to an AI down-scaling DNN.

The DNN setting information updater 1336 may obtain AI data related toAI down-scaling of the at least one original image to the first image.The DNN setting information updater 1336 may obtain DNN settinginformation for performing AI up-scaling, based on AI data. The DNNsetting information updater 1336 may update DNN setting informationbased on the DNN setting information obtained based on the AI data.

In this case, the DNN setting information may be DNN information that isupdated for performing AI up-scaling of at least one second imagecorresponding to at least one original image via joint training of theup-scaling DNN and the down-scaling DNN used for AI down-scaling of theat least one original image or separate training of the up-scaling DNNby using the at least one original image as a training image.

It has been described above with reference to FIG. 12 that the AIencoding apparatus 1200 updates DNN setting information for performingAI up-scaling of the at least one second image corresponding to the atleast one original image via joint training of the up-scaling DNN andthe down-scaling DNN used for AI down-scaling of the at least oneoriginal image or separate training of the up-scaling DNN by using theat least one original image as a training image. Because the AI decodingapparatus 1300 uses parameters of the AI up-scaling DNN, which areupdated using an original image as a training image, instead ofparameters of the AI up-scaling DNN previously determined by using aseparately provided image as a training image, the AI decoding apparatus1300 may optimally perform AI up-scaling on the second image.

The DNN setting information may include at least one of the number ofconvolution layers, the number of filter kernels in each of theconvolution layers, or information about a size of each filter kernel.

Furthermore, the DNN setting information may include weights of at leastone filter kernel in at least one convolutional layer constituting theup-scaling DNN. Embodiments of the disclosure are not limited thereto,and the DNN setting information may include a bias for at least oneoutput channel.

The DNN setting information updater 1336 may obtain a first flag basedon AI data and determine, based on the first flag, whether the secondimage is to be up-scaled by using predetermined DNN setting informationof the AI up-scaling DNN or by using DNN setting information of the AIup-scaling DNN, which is updated using an original image as a trainingimage. The first flag may be a flag indicating that the second image isto be up-scaled by using the predetermined DNN setting information ofthe AI up-scaling DNN or based on the DNN setting information of the AIup-scaling DNN, which is updated using the original image as a trainingimage. When the first flag has a first value, the first flag mayindicate that the second image is to be up-scaled by using predeterminedDNN setting information of the AI up-scaling DNN trained as describedabove with reference to FIG. 9, wherein the predetermined DNN settinginformation is stored in the AI decoding apparatus 1300.

In this case, the DNN setting information updater 1336 may determine apiece of DNN setting information of an AI up-scaling DNN from amongpieces of predetermined DNN setting information of a plurality of AIup-scaling DNNs and determine the piece of DNN setting information ofthe AI up-scaling DNN as DNN setting information for performing AIup-scaling on the second image.

When the first flag has a second value, the first flag may indicate thatthe second image is to be up-scaled based on DNN setting information ofthe AI up-scaling DNN, which is updated using an original image as atraining image.

In this configuration, the DNN setting information updater 1336 maydetermine the DNN setting information of the AI up-scaling DNN, which isupdated using the original image as a training image, as DNN settinginformation for performing AI up-scaling on the second image.

The first or second value may be 0 or 1, but the value is not limitedthereto.

In this configuration, the DNN setting information of the AI up-scalingDNN, which is updated using the original image as a training image, maybe generated via model compression. The AI encoding apparatus 1200 mayencode DNN setting information of an up-scaling DNN, which is obtainedvia joint training of a down-scaling DNN and the up-scaling DNN orseparate training of the up-scaling DNN by using an original image as atraining image, and the DNN setting information updater 1336 may obtaininformation about the encoded DNN setting information.

For example, the DNN setting information updater 1336 may obtaininformation about a structure of an up-scaling DNN (e.g., the number ofconvolution layers, the number of filter kernels in each convolutionlayer, and information about a size of each filter kernel). The DNNsetting information updater 1336 may obtain information about the numberof convolution layers in an up-scaling DNN, the number of filter kernelsin each convolution layer, and a size of each filter kernel and updatethe number of convolution layers in the up-scaling DNN, the number offilter kernels in each convolution layer, and the size of each filterkernel based on the obtained information.

The DNN setting information updater 1336 may obtain pieces of DNNsetting information for a plurality of up-scaling DNN structures,determine a DNN structure from among the plurality of up-scaling DNNstructures based on at least one of a bitrate of the image data or aresolution of the original image or the first image, and update thenumber of convolutional layers in an up-scaling DNN, the number offilter kernels in each convolution layer, and a size of each filterkernel based on DNN setting information corresponding to the determinedDNN structure.

Alternatively, the DNN setting information updater 1336 may obtainpieces of DNN setting information for a plurality of DNN structures,determine one DNN structure from among the plurality of DNN structuresbased on at least one available DNN structure, and update the number ofconvolutional layers in an up-scaling DNN, the number of filter kernelsin each convolution layer, and a size of each filter kernel based on DNNsetting information corresponding to the determined DNN structure. Inthis case, the at least one available DNN structure may be determinedaccording to the capability of a processor through which the AIup-scaler 1334 is implemented in the AI decoding apparatus 1300. Forexample, the DNN setting information updater 1336 may obtain informationabout a weight residual or a bias residual in the up-scaling DNN. Inthis case, the weight residual or bias residual may mean a differencebetween a weight or a bias for performing up-scaling via the up-scalingDNN in an immediately preceding period unit and a weight or a bias forperforming up-scaling via the up-scaling DNN in a current period unit.In this case, encoded information about the weight residual or the biasresidual in the up-scaling DNN is information encoded using variousentropy encoding techniques such as Differential Pulse Code Modulation(DPCM), Run Length Coding (RLC), and Huffman coding. The DNN settinginformation updater 1336 may obtain information about the weightresidual or the bias residual in the up-scaling DNN by performing aninverse encoding operation.

The DNN setting information updater 1336 may update a weight or a biasfor performing up-scaling via the up-scaling DNN in the current periodunit by adding a weight residual or a bias residual in the up-scalingDNN to a weight or a bias for performing up-scaling via the up-scalingDNN in the immediately preceding period unit. Embodiments of thedisclosure are not limited thereto, and the DNN setting informationupdater 1336 may perform entropy decoding on encoded DNN settinginformation based on a context model. For example, the DNN settinginformation updater 1336 may generate a context model based on DNNsetting information of the up-scaling DNN fora preceding period unit andperform entropy decoding on encoded DNN setting information of theup-scaling DNN by using the generated context model.

The context model may be updated each time weights or a bias of a filterkernel in the up-scaling DNN for the current period unit is decoded, andinformation about next weights or biases of a filter kernel in theup-scaling DNN for the current period unit may be entropy-decoded basedon the updated context model. Examples of entropy decoding based oncontexts may include range coding, arithmetic decoding, and Huffmandecoding, but the entropy decoding is not limited thereto. The contextmodel may be determined for the entire up-scaling DNN, but the contextmodel is not limited thereto, and the context model may be determinedfor each convolutional layer, for at least one filter kernelcorresponding to one output channel in a convolutional layer, and foreach filter kernel in the convolutional layer.

The DNN setting information updater 1336 may decode weight informationor bias information for all layers of the up-scaling DNN and updateweights or biases in all the layers based on the decoded weightinformation or bias information, but the DNN setting information updater1336 is not limited thereto. The DNN setting information updater 1336may decode weight information or bias information for some of the layersand update weights or biases in the corresponding layers. Alternatively,the DNN setting information updater 1336 may decode weight informationor bias information of some filter kernels in a layer and update weightsor biases in the corresponding filter kernels in the layer. For example,the DNN setting information updater 1336 may decode weightinformation/bias information in a last layer of the up-scaling DNN andupdate weights or biases in the last layer.

The DNN setting information updater 1336 may decode weight residualinformation or bias residual information indicating a difference betweena weight or a bias of all or some of the filter kernels in all or someof the convolution layers in the up-scaling DNN before the weight orbias is updated and a weight or a bias of all or some of the filterkernels in all or some of the convolution layers in the up-scaling DNNafter the weight or bias is updated and then update weights or biases ofall or some of the filter kernels in all or some of the convolutionlayers based on the decoded weight residual information or bias residualinformation. The DNN setting information updater 1336 may decode weightresidual information and obtain a weight residual by performing inversefrequency transformation on the decoded weight residual information.

The DNN setting information updater 1336 may perform entropy decoding oninformation about weights or biases in an up-scaling DNN used forperforming AI up-scaling on the second image based on a context model,but the DNN setting information updater 1336 is not limited thereto. TheDNN setting information updater 1336 may entropy-decode, based on thecontext model, a difference between a weight or a bias updated for acurrent period unit and a weight or a bias used for up-scaling in animmediately preceding period unit. In this case, the context model maybe based on information about a difference between weights or biasesdetermined for up-scaling at least one of an immediately preceding imageor an image preceding the immediately preceding image. The encoded DNNsetting information may be generated by entropy-encoding weightinformation or bias information in combination with a model compressiontechnique (e.g., pruning or quantization).

The DNN setting information updater 1336 may periodically update DNNsetting information of the up-scaling DNN. For example, the DNN settinginformation updater 1336 may decode encoded DNN setting information foreach GOP, each IRAP period (e.g., each IDR picture period), eachsequence, or each unit of a preset number of frames, and update DNNsetting information of the up-scaling DNN based on the decoded DNNsetting information.

The DNN setting information updater 1336 may update DNN settinginformation whenever necessary. For example, the DNN setting informationupdater 1336 may decode first flag information encoded for each presetperiod, determine whether to update DNN setting information of theup-scaling information based on the decoded first flag information, andupdate the DNN setting information when the DNN setting informationupdater 1336 determines to update the DNN setting information. In thiscase, the encoded first flag information may be flag informationregarding whether the DNN setting information of the up-scaling DNN isto be updated.

When the first flag information has a first value, the DNN settinginformation updater 1336 may determine not to update the DNN settinginformation of the up-scaling DNN.

When the first flag information has a second value, the DNN settinginformation updater 1336 may determine to update the DNN settinginformation of the up-scaling DNN.

The first or second value may be 0 or 1, but the value is not limitedthereto.

The DNN setting information updater 1336 may parse the first flaginformation for each preset period and when the first flag informationis not parsed for each preset period, and determine not to update DNNsetting information of the up-scaling DNN.

The DNN setting information updater 1336 may decode encoded second flaginformation and determine, based on the decoded second flag information,whether a second image is to be up-scaled by using predetermined DNNsetting information of an AI up-scaling DNN or by using DNN settinginformation of the AI up-scaling DNN, which is updated using an originalimage as a training image. In this case, the encoded second flaginformation may be flag information indicating whether the second imageis to be up-scaled based on the predetermined DNN setting information ofthe AI up-scaling DNN or based on the DNN setting information of the AIup-scaling DNN, which is updated using the original image as a trainingimage.

When the second flag information has a first value, the DNN settinginformation updater 1336 may determine that the second image is to beup-scaled by using the predetermined DNN setting information of theup-scaling DNN.

When the second flag information has a second value, the DNN settinginformation updater 1336 may determine that the second image is to beup-scaled by using the DNN setting information of the up-scaling DNN,which is updated using the original image as a training image.

The DNN setting information updater 1336 may parse the second flaginformation for each preset period and when the second flag informationis not parsed for each preset period, and determine that the secondimage is to be up-scaled by using the predetermined DNN settinginformation of the up-scaling DNN.

Although it has been described above that the DNN setting informationupdater 1336 decodes encoded DNN setting information of the up-scalingDNN, embodiments of the disclosure are not limited thereto, and it willbe understood by those of ordinary skill in the art that the parser 1314may perform the above-described decoding operation of the DNN settinginformation updater 1336 and transmit the decoded DNN settinginformation to the DNN setting information updater 1336 via theoutputter 1316.

The AI up-scaler 1334 may generate a third image by performingup-scaling on the second image via the up-scaling DNN operatingaccording to the DNN setting information. The DNN setting informationmay be received from the DNN setting information updater 1336.

FIG. 14A is a flowchart of an AI encoding method according to anembodiment of the disclosure.

The AI encoding apparatus 1200 generates up-scaling DNN settinginformation for performing AI up-scaling corresponding to AIdown-scaling, which is DNN information updated via DNN joint or separatetraining using an original image (operation S1405). A process, performedby the AI encoding apparatus 1200, of generating DNN setting informationupdated via DNN joint or separate training using an original image willbe described in detail below with reference to FIGS. 16A, 16B, and 17A.

The AI encoding apparatus 1200 may obtain image data generated byperforming first encoding on a first image obtained by performing AIdown-scaling on the original image (operation S1410).

The AI encoding apparatus 1200 transmits image data and AI dataincluding up-scaling DNN setting information (operation S1415).

FIG. 14B is a flowchart of an AI encoding method via DNN joint trainingbased on an original image, according to an embodiment of thedisclosure.

The AI encoding apparatus 1200 may obtain a first training image byperforming AI down-scaling on an original image via a down-scaling DNNand obtain a third training image by performing AI up-scaling on thefirst training image via an up-scaling DNN (operation S1420).

The AI encoding apparatus 1200 may update DNN information via DNN jointtraining based on the original image, the first training image, and thethird training image (operation S1425).

The AI encoding apparatus 1200 may obtain a first image by performing AIdown-scaling on original image via a down-scaling DNN based on theupdated DNN information and obtain the third training image byperforming, via an up-scaling DNN, up-scaling on a second training imagegenerated by performing first encoding and first decoding on the firsttraining image (operation S1430).

The AI encoding apparatus 1200 may generate up-scaling DNN settinginformation for performing AI up-scaling corresponding to AIdown-scaling by updating the DNN information via training of theup-scaling DNN based on the original image and the third training image(operation S1435).

The AI encoding apparatus 1200 may obtain image data generated byperforming first encoding on the first image generated by performing thedown-scaling on the original image (operation S1440). In this case, theAI down-scaling may be performed based on DNN information set via DNNjoint training using the original image.

The AI encoding apparatus 1200 may generate AI encoding data includingthe image data and AI data containing the up-scaling DNN settinginformation (operation S1445).

A process, performed by the AI encoding apparatus 1200, of generatingDNN setting information updated via DNN joint training using an originalimage will be described in detail below with reference to FIGS. 16A and16B.

FIG. 14C is a flowchart of an AI encoding method via DNN separatetraining based on an original image, according to an embodiment of thedisclosure.

The AI encoding apparatus 1200 may obtain a first training image byperforming AI down-scaling on an original image via a down-scaling DNNand obtain a third training image by performing, via an up-scaling DNN,AI up-scaling on a second image generated by performing first encodingand first decoding on the first training image (operation 1450).

The AI encoding apparatus 1200 may update DNN information via separatetraining of the up-scaling DNN based on the original image and the thirdtraining image and generate up-scaling DNN setting information forperforming AI up-scaling corresponding to AI down-scaling (operationS1455).

The AI encoding apparatus 1200 may obtain image data generated byperforming first encoding on a first image generated by performingdown-scaling on the original image (operation S1460).

The AI encoding apparatus 1200 may generate AI encoding data includingthe image data and AI data containing the up-scaling DNN settinginformation (operation S1465).

A process, performed by the AI encoding apparatus 1200, of generatingDNN setting information updated via DNN separate training using anoriginal image will be described in detail below with reference to FIG.17A.

FIG. 15 is a flowchart of an AI decoding method according to anembodiment of the disclosure.

The AI decoding apparatus 1300 obtains image data and AI data (operationS1510).

The AI decoding apparatus 1300 obtains a second image based on the imagedata (operation S1520). The AI decoding apparatus 1300 may obtain thesecond image by performing first decoding on the image data.

The AI decoding apparatus 1300 obtains, from AI data, up-scaling DNNsetting information for performing AI up-scaling, which is DNNinformation updated via DNN joint or separate training using theoriginal image (operation S1530).

A process, performed by the AI decoding apparatus 1300, of obtaining,from AI data, up-scaling DNN setting information for performing AIup-scaling, which is DNN information updated via DNN joint or separatetraining using an original image, will be described in detail below withreference to FIGS. 16C and 17B.

The AI decoding apparatus 1300 obtains a third image by performingup-scaling on the second image based on the up-scaling DNN settinginformation (operation S1540).

FIG. 16A is a diagram for describing, as a first stage of jointtraining, a process, performed by the AI encoding apparatus 1200, ofdetermining pieces of optimal DNN setting information of a down-scalingDNN and an up-scaling DNN via joint training of the down-scaling DNN andthe up-scaling DNN by using an original image as a training image,according to another embodiment of the disclosure.

Referring to FIG. 16A, the AI encoding apparatus 1200 may generate afirst training image 1602 as a result of performing AI down-scalingbased on a first DNN 700 by using an original image 1601 as a trainingimage. In this case, the original image 1601 may be frames in a currentGOP unit, a current IRAP period unit, a current sequence unit, or acurrent unit of a predetermined number of frames to be encoded into oneor more frames, but the original image 1601 is not limited thereto.Furthermore, initial DNN setting information of the first DNN 700 andinitial DNN setting information of the second DNN 300 may berespectively used for down-scaling and up-scaling an image in animmediately preceding period unit. Alternatively, the initial DNNsetting information of the first or second DNN 700 or 300 may be one ofa plurality of pieces of predetermined DNN setting information or DNNsetting information set as default.

However, embodiments of the disclosure are not limited thereto, andinitial weights in the first and second DNNs 700 and 300 may be valuesobtained with random sampling to follow a particular probabilitydistribution. Initial biases in the first and second DNNs 700 and 300may be determined to be zero.

The AI encoding apparatus 1200 may generate second loss information 1620based on the complexity of the first training image 1602. In this case,the second loss information 1620 may correspond to the second lossinformation described above with reference to FIG. 12.

The AI encoding apparatus 1200 may generate a third training image 1604as a result of performing AI up-scaling based on the second DNN 300 byusing the first training image 1602.

The AI encoding apparatus 1200 may generate first loss information 1610and third loss information 1630, each corresponding to a result ofcomparing information related to the original image 1601 withinformation related to the third training image 1604. In this case, thefirst loss information 1610 and the third loss information 1630 mayrespectively correspond to the first loss information and the third lossinformation described above with reference to FIG. 12.

The AI encoding apparatus 1200 may determine a loss value Loss_(DS)based on the first loss information 1610, the second loss information1620, and the third loss information 1630 according to Equation 6 below.

Loss_(DS) =a*FIRST LOSS INFORMATION+b*SECOND LOSS INFORMATION+c*THIRDLOSS INFORMATION  [Equation 6]

The AI encoding apparatus 1200 may update pieces of DNN settinginformation of the first DNN 700 and the second DNN 300 based on theloss value Loss_(DS).

The AI encoding apparatus 1200 may perform joint training of the firstand second DNNs 700 and 300 by iteratively repeating the above-describedprocess and determine optimal DNN setting information of the first andsecond DNNs 700 and 300 via joint training based on the loss valueLoss_(DS).

FIG. 16B is a diagram for describing, as a second stage of jointtraining, a process by which the AI encoding apparatus 1200 selects DNNsetting information of a down-scaling DNN, which is determined accordingto the process illustrated in FIG. 16A, determines optimal DNN settinginformation of an up-scaling DNN via separate training of the up-scalingDNN, and transmits the optimal DNN setting information of the up-scalingDNN through a bitstream, according to an embodiment of the disclosure.

Referring to FIG. 16B, the AI encoding apparatus 1200 selects theoptimal DNN setting information determined via the joint training of thefirst and second DNNs 700 and 300 as described above with reference toFIG. 16A. In this case, initial DNN setting information of the secondDNN 300 may be determined based on the optimal DNN setting informationof the second DNN 300 determined via the joint training of the first andsecond DNNs 700 and 300, but the initial DNN setting information is notlimited thereto. An initial weight in the second DNN 300 may be a valuerandomly sampled based on a particular probability distribution, and aninitial bias in the second DNN 300 may be 0.

The AI encoding apparatus 1200 may generate a first training image 1602as a result of performing AI down-scaling based on the first DNN 700 byusing an original image 1601 as a training image. The AI encodingapparatus 1200 may perform first encoding 1603 and first decoding 1605on the first training image 1602. The AI encoding apparatus 1200 maygenerate a third training image 1604 by performing AI up-scaling on animage generated via the first decoding 1605 based on the second DNN 300.The AI encoding apparatus 1200 may generate first loss information 1610and third loss information 1630, each corresponding to a result ofcomparing image information related to the original image 1601 withimage information related to the third training image 1604.

The AI encoding apparatus 1200 may determine a loss value Loss_(US)based on the first loss information 1610 and the third loss information1630 according to Equation 7 below.

Loss_(US) =d*FIRST LOSS INFORMATION+e*THIRD LOSS INFORMATION  [Equation7]

The AI encoding apparatus 1200 may update DNN setting information of thesecond DNN 300 based on the loss value Loss_(US). In this case, the DNNsetting information of the second DNN 300 may be updated via modelcompression.

The AI encoding apparatus 1200 may perform separate training of thesecond DNN 300 by repeating the above-described process and determineoptimal DNN setting information of the second DNN 300 via separatetraining based on the loss value Loss_(US).

The AI encoding apparatus 1200 may encode the determined optimal DNNsetting information of the second DNN 300, and the transmitter 1230 ofthe AI encoding apparatus 1200 may generate AI encoding data includingthe encoded DNN setting information and image data generated via thefirst encoding 1603. In this case, the DNN setting information mayinclude weight residual information or bias residual information of thesecond DNN 300. Furthermore, the AI encoding data may be represented inthe form of a bitstream.

FIG. 16C is a diagram for describing a process by which the AI decodingapparatus 1300 performs AI up-scaling on a second image via anup-scaling DNN based on DNN setting information of the up-scaling DNN,which is included in AI encoding data, according to an embodiment of thedisclosure.

As described above with reference to FIG. 16B, the AI encoding apparatus1200 may generate AI encoding data.

Referring to FIG. 16C, the receiver 1310 of the AI decoding apparatus1300 receives AI encoding data.

As described above, the AI encoding data may be represented as abitstream. The receiver 1310 of the AI decoding apparatus 1300 obtainsimage data and DNN setting information from the AI encoding data. TheDNN setting information updater 1336 of the AI decoding apparatus 1300may configure the second DNN 300 based on the DNN setting information.The AI decoding apparatus 1300 may obtain the image data from the AIencoding data and generate a second image 1670 by performing firstdecoding 1660 on the image data.

The AI decoding apparatus 1300 may generate a third image 1680 byup-scaling the second image 1670 via the second DNN 300 based on the DNNsetting information. In this case, because the DNN setting informationis DNN setting information of the second DNN 300 optimized using anoriginal image as a training image as described above with reference toFIGS. 16A and 16B, a quality of the third image 1680 may be improvedcompared to a quality of a third image generated by up-scaling a secondimage based on DNN setting information of a second DNN, which ispredetermined based on a training image separately provided by the AIencoding apparatus 1200 and the AI decoding apparatus 1300.

FIG. 17A is a diagram for describing a process by which the AI encodingapparatus selects DNN setting information of a down-scaling DNN,determines optimal DNN setting information of an up-scaling DNN viaseparate training of the up-scaling DNN, and transmits the optimal DNNsetting information of the up-scaling DNN in a bitstream, according toan embodiment of the disclosure.

Referring to FIG. 17A, the AI encoding apparatus 1200 selects weights ofthe first DNN 700. In this case, selected DNN setting information of thefirst DNN 700 may be setting information used for down-scaling anoriginal image in an immediately preceding period unit. However,embodiments of the disclosure are not limited thereto, and the DNNsetting information of the first DNN 700 may be one of a plurality ofpieces of predetermined DNN setting information or DNN settinginformation set as default.

Moreover, initial DNN setting information of the second DNN 300 may beDNN setting information used for up-scaling an original image in animmediately preceding unit. However, embodiments of the disclosure arenot limited thereto, and an initial weight in the second DNN 300 may bea value randomly sampled based on a particular probability distribution,and an initial bias in the second DNN 300 may be 0.

The AI encoding apparatus 1200 may generate a first training image 1702as a result of performing AI down-scaling based on the first DNN 700 byusing an original image 1701 as a training image. The AI encodingapparatus 1200 may perform first encoding 1703 and first decoding 1705on the first training image 1702. The AI encoding apparatus 1200 maygenerate a third training image 1704 by performing AI up-scaling on animage generated via the first decoding 1705 based on the second DNN 300.The AI encoding apparatus 1200 may generate first loss information 1710and third loss information 1730, each corresponding to a result ofcomparing image information related to the original image 1701 withimage information related to the third training image 1704.

The AI encoding apparatus 1200 may determine a loss value Loss_(US)based on the first loss information 1710 and the third loss information1730 according to Equation 8 below.

Loss_(US) =d*FIRST LOSS INFORMATION+e*THIRD LOSS INFORMATION  [Equation8]

The AI encoding apparatus 1200 may update DNN setting information of thesecond DNN 300 based on the loss value Loss_(US). In this case, the DNNsetting information of the second DNN 300 may be updated via modelcompression.

The AI encoding apparatus 1200 may perform separate training of thesecond DNN 300 by repeating the above-described process and determineoptimal DNN setting information of the second DNN 300 via separatetraining based on the loss value Loss_(US).

The AI encoding apparatus 1200 may encode the determined optimal DNNsetting information of the second DNN 300, and the transmitter 1230 ofthe AI encoding apparatus 1200 may generate AI encoding data includingthe encoded DNN setting information and image data generated via thefirst encoding 1703. In this case, the DNN setting information mayinclude weight residual information or bias residual information of thesecond DNN 300.

FIG. 17B is a diagram for describing a process, performed by the AIdecoding apparatus 1300, of performing AI up-scaling on a second imagevia an up-scaling DNN based on DNN setting information of the up-scalingDNN, which is included in AI encoding data, according to an embodimentof the disclosure.

As described above with reference to FIG. 17A, the AI encoding apparatus1200 may generate AI encoding data.

Referring to FIG. 17B, the receiver 1310 of the AI decoding apparatus1300 receives AI encoding data. As described above, the AI encoding datamay be represented as a bitstream. The receiver 1310 of the AI decodingapparatus 1300 obtains image data and DNN setting information from theAI encoding data. The DNN setting information updater 1336 of the AIdecoding apparatus 1300 may determine the second DNN 300 based on theDNN setting information.

The AI decoding apparatus 1300 may generate a second image 1770 byperforming first decoding 1760 on the image data. The AI decodingapparatus 1300 may generate a third image 1780 by up-scaling the secondimage 1770 via the second DNN 300 based on the DNN setting information.In this case, because the DNN setting information is DNN settinginformation of the second DNN 300 optimized using an original image as atraining image as described above with reference to FIG. 17A, a qualityof the third image 1780 may be improved compared to a quality of a thirdimage generated by up-scaling a second image based on DNN settinginformation of a second DNN, which is predetermined based on aseparately provided training image.

FIG. 18 is a flowchart of a process, performed by the AI decodingapparatus 1300, of up-scaling a second image by updating DNN settinginformation of an up-scaling DNN, which is predetermined based on flagsobtained from AI encoding data or of up-scaling the second image byupdating DNN setting information of the up-scaling DNN, which isoptimized for an original image, according to an embodiment of thedisclosure.

Referring to FIG. 18, the AI decoding apparatus 1300 may determine,based on a value of a first flag, whether the first flag indicateswhether to use predetermined DNN setting information of an up-scalingDNN (operation S1805).

When the first flag indicates that the predetermined DNN settinginformation of the up-scaling DNN is to be used, the AI decodingapparatus 1300 may up-scale an image by using the predetermined DNNsetting information of the up-scaling DNN (operation S1810). In thiscase, the predetermined DNN setting information of the up-scaling DNNmay be stored in the AI decoding apparatus 1300.

The AI decoding apparatus 1300 may periodically obtain a second flag anddetermine, based on a value of the second flag, whether the second flagindicates whether to use DNN setting information of the up-scaling DNN,other than previous DNN setting information among a plurality of piecesof predetermined DNN setting information of the up-scaling DNN(operation S1815).

When the second flag indicates that the DNN setting information of theup-scaling DNN, other than the previous DNN setting information amongthe plurality of pieces of predetermined DNN setting information of theup-scaling DNN, is to be used, the AI decoding apparatus 1300 may changeDNN setting information to be used for the up-scaling DNN to DNN settinginformation of the up-scaling DNN, other than that used in a previousperiod among the plurality of pieces of predetermined DNN settinginformation (operation S1820).

When the second flag indicates that the DNN setting information of theup-scaling DNN, other than the previous DNN setting information amongthe plurality of pieces of predetermined DNN setting information of theup-scaling DNN, is not to be used, the AI decoding apparatus 1300 maydetermine that DNN setting information used in the previous period ismaintained as DNN setting information to be used for the up-scaling DNN(operation S1825).

The AI decoding apparatus 1300 may up-scale the image by usingpredetermined DNN setting information of the up-scaling DNN (operationS1830). In this case, the predetermined DNN setting information of theup-scaling DNN may be the DNN setting information changed in operationS1820 or DNN setting information maintained in operation S1825.

When the first flag indicates that the predetermined DNN settinginformation of the up-scaling DNN is not to be used, the AI decodingapparatus 1300 may up-scale the image by using DNN setting informationof the up-scaling DNN optimized for an original image (operation S1835).Descriptions with respect to the DNN setting information of theup-scaling DNN optimized for the original image are already providedabove with reference to FIGS. 16A through 17B, and thus, a redundantdescription is not repeated below.

The AI decoding apparatus 1300 may periodically obtain a third flag anddetermine, based on a value of the third flag, whether the third flagindicates whether to update DNN setting information with DNN settinginformation of the up-scaling DNN optimized fora current period(operation S1840).

When the third flag indicates that the DNN setting information is to beupdated, the AI decoding apparatus 1300 may change DNN settinginformation to be used for the up-scaling DNN to DNN setting informationof the up-scaling DNN optimized for the current period of the originalimage (operation S1845).

When the third flag indicates that the DNN setting information is not tobe updated, the AI decoding apparatus 1300 may determine to maintain DNNsetting information of the up-scaling DNN used in a previous period(operation S1850). The AI decoding apparatus 1300 may up-scale the imageby using the DNN setting information of the up-scaling DNN optimized forthe original image (operation S1855). In this case, the DNN settinginformation optimized for the original image may be the DNN settinginformation changed in S1845 or DNN setting information maintained inS1850.

FIG. 19A illustrates examples of default weights and biases, weights andbiases in an up-scaling DNN, which are optimized for an original image,and weight differences and bias differences in the up-scaling DNN,according to an embodiment of the disclosure.

Referring to FIG. 19A, the AI encoding apparatus 1200 may determinedefault weight matrices 1910 of size 3×3 and default bias matrices 1920of size 1×1 of a filter kernel in a convolution layer of an up-scalingDNN. In this case, the default weight matrices 1910 and the default biasmatrices 1920 may be weights and biases previously determined by using aseparate training image in the AI encoding apparatus 1200, but theweights and biases are not limited thereto. The weights and biases maybe weights and biases determined by using an original image as atraining image in a immediately preceding period unit.

According to Equation 9 below, an output matrix Y_(L,t, d_out) may begenerated using an input matrix X_(L,t, d_in), default weight matricesW_(L,t,d_in,d_out) (e.g., 1910), and the default bias matricesB_(L,t,d_out) (e.g., 1920). In Equation 9, activation( ) andconvolution( ) denote an activation function and a convolution operationfunction, respectively, L, d_in, and d_out denote a convolutional layer,an input channel (depth) of layer L, and an output channel of layer L,respectively, and t denotes the time.

$\begin{matrix}{Y_{L,t,d_{out}} = {{activation}\left( {{\sum\limits_{d_{in}}{{convolution}\left( {X_{L,t,d_{in},}W_{L,t,d_{in},d_{out}}} \right)}} + B_{L,t,d_{out}}} \right)}} & \left\lbrack {{Equation}\mspace{14mu} 9} \right\rbrack\end{matrix}$

Referring to FIG. 19A, the AI encoding apparatus 1200 may determineoptimal weight matrices 1930 of size 3×3 and optimal bias matrices 1940of size 1×1 by using a current original image as a training image.Values of elements in the optimal weight matrices 1930 and the optimalbias matrices 1940 may be different from those of elements of thedefault weight matrices 1910 and the default bias matrices 1920,respectively.

The AI encoding apparatus 1200 may entropy-encode the optimal weightmatrices 1930 and the optimal bias matrices 1940 based on a contextmodel for transmission.

The AI encoding apparatus 1200 may determine weight difference matrices1950 of size 3×3 and bias difference matrices 1960 of size 1×1 based onthe default weight matrices 1910, the default bias matrices 1920, theoptimal weight matrices 1930, and the optimal bias matrices 1940.

A value of an element in each of the weight difference matrices 1950 maybe a difference between a value of an element in a corresponding one ofthe default weight matrices 1910 and a value of an element in thecorresponding optimal weight matrix 1930, and a value of an element ofeach of the bias difference matrices 1960 may be a difference between avalue of an element in a corresponding one of the default bias matrices1920 and a value of an element in the corresponding optimal bias matrix1940.

The AI encoding apparatus 1200 may entropy-encode the weight differencematrices 1950 and the bias difference matrices 1960 for transmission.

FIG. 19B illustrates examples of weights and biases in an up-scalingDNN, which are optimized for an original image, and weights and biasesin the up-scaling DNN, which are determined via quantization andpruning, according to an embodiment of the disclosure.

Referring to FIG. 19B, the AI encoding apparatus 1200 may determineoptimal weight matrices 1970 of size 3×3 and optimal bias matrices 1975of size 1×1 by using a current original image as a training image.

The AI encoding apparatus 1200 may determine weight matrices 1980 ofsize 3×3 by performing pruning on the optimal weight matrices 1970. Inother words, referring to FIG. 19B, the AI encoding apparatus 1200 maydetermine the weight matrices 1980 by changing to zero all values ofelements in the optimal weight matrices 1970 that are less than anabsolute value of 0.02. However, an absolute value used to change avalue of an element to zero is not limited to 0.02, and it will beunderstood by those of ordinary skill in the art that a value of anelement that is less than a value close to 0 may be changed to 0.

The AI encoding apparatus 1200 may determine weight matrices 1990 ofsize 3×3 and bias matrices 1995 of size 1×1 by respectively performingquantization on the weight matrices 1980 and the optimal bias matrices1975. In other words, referring to FIG. 19B, the AI encoding apparatus1200 may determine the weight matrices 1990 by performing amultiplication operation in which values of elements in the weightmatrices 1980 are multiplied element-wise by 128 and a roundingoperation on the element-wise products and determine the bias matrices1995 by performing a rounding operation on values of elements in theoptimal bias matrices 1975. However, embodiments of the disclosure arenot limited thereto, and those of ordinary skill in the art willappreciate that may understand that quantization of various embodimentsof the disclosure may be performed on weight/bias matrices.

FIG. 20A is a diagram for describing a method of encoding weights in anup-scaling DNN, which are optimized for an original image, according toan embodiment of the disclosure.

Referring to FIG. 20A, the AI encoding apparatus 1200 may determine aweight difference matrix 2010 of a 4λ4 filter kernel. In this case,weight difference values in the weight difference matrix 2010 are weightdifference values in an up-scaling DNN, which are optimized for anoriginal image, as described above with reference to FIG. 19A, and adetailed description thereof will be omitted below.

The AI encoding apparatus 1200 may determine a weight matrix 2015 byperforming pruning on the weight difference matrix 2010 of the 4λ4filter kernel.

The AI encoding apparatus 1200 may determine a weight matrix 2020 byperforming quantization on the weight matrix 2015. Descriptions withrespect to pruning and quantization are already provided above withreference to FIG. 19B, and thus, will not be repeated.

The AI encoding apparatus 1200 may then generate AI data 2040 byperforming RLC 2030 on weights in the weight matrix 2020. In this case,the AI data 2040 may be represented in the form of a bitstream.

FIG. 20B is a diagram for describing a method of encoding weights in anup-scaling DNN, which are optimized for an original image, according toanother embodiment of the disclosure.

Referring to FIG. 20B, the AI encoding apparatus 1200 may determine aweight difference matrix 2055 of a 4λ4 filter kernel. In this case,weight difference values in the weight difference matrix 2055 are weightdifference values in an up-scaling DNN, which are optimized for anoriginal image, as described above with reference to FIG. 19A, and aredundant description thereof is omitted below.

The AI encoding apparatus 1200 may determine a weight matrix 2060 byperforming transformation (e.g., discrete cosine transform) on theweight difference matrix 2055 of the 4λ4 filter kernel.

The AI encoding apparatus 1200 may determine a weight matrix 2065 byperforming pruning on the weight matrix 2060.

The AI encoding apparatus 1200 may determine a weight matrix 2070 byperforming quantization on the weight matrix 2065.

The AI encoding apparatus 1200 may generate AI data 2090 by performingRLC 2080 on the weights of the weight matrix 2070.

The AI decoding apparatus 1300 may perform run-length decoding on the AIdata 2090 to generate a two-dimensional (2D) weight matrix, and restorea weight difference matrix via inverse quantization and inversetransformation. In this case, the restored weight difference matrix maybe different from the weight difference matrix 2055. The mismatch iscaused by data loss due to the pruning and quantization.

According to the method of FIG. 20B, the AI encoding apparatus 1200 mayperform transformation on the weight difference matrix 2055 such thatdata values may concentrate at low frequencies while data values at highfrequencies may be closer to zero. Thus, unnecessary information may bediscarded more effectively during subsequent pruning and quantizationthan when using the method described with reference to FIG. 20A. The AIdecoding apparatus 1300 may perform inverse transformation correspondingto the transformation performed by the AI encoding apparatus 1200.

In this case, the AI decoding apparatus 1300 may generate a 2D weightmatrix by performing run-length decoding on the AI data 2090 and restorea weight difference matrix by performing inverse quantization andinverse transformation on the 2D weight matrix.

FIG. 21A is a diagram for describing a process, performed by the AIencoding apparatus 1200 of entropy-encoding weight information of anup-scaling DNN, which is optimized for an original image, based on acontext model, according to an embodiment of the disclosure, and FIG.21B is a diagram for describing a process, performed by the AI decodingapparatus 1300, of entropy-decoding weight information of the up-scalingDNN, which is optimized for the original image, based on the contextmodel, according to an embodiment of the disclosure.

Referring to FIG. 21A, in the AI encoding apparatus 1200, a contextmodel generator 2110 may generate a context model M_(t-1) based on aweight w_(t-1) that is a weight of an image at a previous time _(t-1),and an entropy encoder 2120 may generate weight information byentropy-encoding a weight w_(t), which is a weight of an image at acurrent time t, based on the context model M_(t-1). In this case,entropy encoding may be performed in combination with model compressiontechniques such as pruning and quantization. In this case, the image atthe previous time t-1 may mean a frame immediately preceding a currentframe at time t, but the time is not limited thereto, and the time maymean a period unit immediately preceding a current period unit includinga frame at time t. In this case, the period unit may be one of a presetnumber of frames, frames in a GOP, frames in a sequence, and frames inan IRAP period, but is not limited thereto.

Referring to FIG. 21B, in the AI decoding apparatus 1300, a contextmodel generator 2130 may generate a context model M_(t-1) based on aweight w_(t-1) that is a restored weight of an image at a previous time_(t-1) in the same manner as in the AI encoding apparatus 1200, and anentropy decoder 2140 may obtain a weight w_(t) that is a weight of animage at a current time t by entropy-decoding the weight informationreceived from the AI encoding apparatus 1200 based on the context modelM_(t-1). In this case, when the entropy encoding is performed incombination with model compression techniques such as pruning andquantization, the AI decoding apparatus 1300 may perform an operationcorresponding to the model compression techniques in combination with anentropy decoding operation. For example, when the entropy encoding isperformed in combination with a quantization operation, the AI decodingapparatus 1300 may perform an inverse quantization operation incombination with an entropy decoding operation.

Moreover, image data and DNN setting information generated by the AIencoding apparatus 1200 and corresponding to at least one quality (e.g.,a quality based on at least one of a resolution or a bitrate) and areprovided to the AI decoding apparatus 1300 through a streaming system.The streaming system is a system including at least one server (e.g., aservice server or a content server) and a terminal, and refers to asystem in which according to a request from the terminal via a network,the server provides image-related data corresponding to the request. Inthis case, the generated image data and DNN setting information may bestored in a content server separate from a service server, and theterminal may receive location information of image data of at least onequality from the service server according to a request and then receiveimage data and DNN setting information from the content servercorresponding to the location information of the image data of the atleast one quality. In this case, the content server is a server forstoring image data and DNN setting information and may be the AIencoding apparatus 1200, but the content server is not limited thereto.The content server may be separate from the AI encoding apparatus 1200,and in this case, the content server may receive the image data and theDNN setting information from the AI encoding apparatus 1200 for storage.

Meanwhile, the embodiments of the disclosure described above may bewritten as computer-executable programs or instructions that may bestored in a medium.

The medium may permanently store the computer-executable programs orinstructions, or store the computer-executable programs or instructionsfor execution or downloading. Also, the medium may be any one of variousrecording media or storage media in which a single piece or plurality ofpieces of hardware are combined, and the medium is not limited to amedium directly connected to a computer system, but may be distributedon a network. Examples of the medium include magnetic media, such as ahard disk, a floppy disk, and a magnetic tape, optical recording media,such as CD-ROM and DVD, magneto-optical media such as a floptical disk,and ROM, RAM, and a flash memory, which are configured to store programinstructions. Other examples of the medium include recording media andstorage media managed by application stores distributing applications orby websites, servers, and the like supplying or distributing othervarious types of software.

Meanwhile, a model related to the DNN described above may be implementedvia software. When the DNN model is implemented via software (forexample, a program module including instructions), the DNN model may bestored in a computer-readable recording medium.

Also, the DNN model may be a part of the AI decoding apparatus 200 or AIencoding apparatus 600 described above by being integrated in a form ofa hardware chip. For example, the DNN model may be manufactured in aform of a dedicated hardware chip for AI, or may be manufactured as apart of an existing general-purpose processor (for example, CPU orapplication processor) or a graphic-dedicated processor (for exampleGPU).

Also, the DNN model may be provided in a form of downloadable software.A computer program product may include a product (for example, adownloadable application) in a form of a software program electronicallydistributed through a manufacturer or an electronic market. Forelectronic distribution, at least a part of the software program may bestored in a storage medium or may be temporarily generated. In thiscase, the storage medium may be a server of the manufacturer orelectronic market, or a storage medium of a relay server.

According to an embodiment of the disclosure, a method and apparatus forperforming AI encoding and AI decoding of an image are capable ofprocessing the image at a low bitrate via AI-based image encoding anddecoding.

Furthermore, according to an embodiment of the disclosure, the methodand apparatus for performing AI encoding and AI decoding of an image mayimprove image quality by performing up-scaling after updating,periodically or whenever necessary, up-scaling DNN setting informationoptimized for an original image.

In addition, according to an embodiment of the disclosure, the methodand apparatus for performing AI encoding and AI decoding of an image mayeffectively reduce the amount of information to be encoded and decodedby effectively signaling DNN setting information for updating DNNsetting information of an up-scaling DNN, optimized for an originalimage, periodically or whenever necessary.

However, it will be appreciated by those of ordinary skill in the artthat the effects that are achievable by the method and apparatus forperforming AI encoding and AI decoding of an image according to anembodiment of the disclosure are not limited to those describedhereinabove and other effects of the disclosure not described hereinwill be more clearly understood from the following description.

While one or more embodiments of the disclosure have been described withreference to the figures, it will be understood by those of ordinaryskill in the art that various changes in form and details may be madetherein without departing from the spirit and scope as defined by thefollowing claims.

What is claimed is:
 1. An electronic device for displaying an image byusing an artificial intelligence (AI), the electronic device comprising:a display; and at least one processor configured to execute one or moreinstructions to: obtain image data generated from performing firstencoding on a first image and AI data related to AI down-scaling of atleast one original image related to the first image; obtain a secondimage corresponding to the first image by performing first decoding onthe image data; obtain, based on the AI data, neural network (NN)setting information for performing AI up-scaling of the second image;identify whether to perform the AI up-scaling by using a filter kernelof a convolution layer in a predetermined NN or whether to perform theAI up-scaling by using a filter kernel of a convolution layer in a NNupdated for performing the AI up-scaling of at least one second imagecorresponding to at least one original image via a joint training of anup-scaling NN and a down-scaling NN used for the AI down-scaling of theat least one original image, the joint training being performed usingthe at least original image; and generate a third image by performingthe AI up-scaling on the second image via the up-scaling NN operatingaccording to the NN setting information based on the identifying.
 2. Theelectronic device of claim 1, wherein the NN setting informationincludes weights and biases of filter kernels in at least oneconvolution layer of the up-scaling NN.
 3. The electronic device ofclaim 1, wherein the processor is further configured to: generate afirst training image via the down-scaling NN by using the at least oneoriginal image, generate a second training image via the up-scaling NNby using the first training image, and update the up-scaling NN and thedown-scaling NN based on first loss information and third lossinformation, the first loss information and the third loss informationcorresponding to a result of comparing the second training image with anoriginal image that has not undergone the AI down-scaling among the atleast one original image, and second loss information generated based onthe first training image.
 4. The electronic device of claim 3, whereinthe first loss information is generated based on a result of comparing aquality parameter of the second training image with a quality parameterof the at least one original image.
 5. The electronic device of claim 4,wherein the third loss information is generated based on a result ofcomparing a feature-related parameter of the second training image witha feature-related parameter of the at least one original image.
 6. Theelectronic device of claim 4, wherein the second loss information isrelated to a spatial complexity of the first training image.
 7. Theelectronic device of claim 1, wherein the processor is furtherconfigured to: generate a first training image via the down-scaling NNby using the at least one original image, perform first encoding on thefirst training image, generate a second training image via theup-scaling NN by using the first training image that has undergone thefirst encoding, and update the up-scaling NN based on first lossinformation and third loss information, the first loss information andthe third loss information corresponding to a result of comparing thesecond training image with an original image that has not undergone theAI down-scaling among the at least one original image.
 8. The electronicdevice of claim 1, wherein the NN setting information updated forperforming the AI up-scaling further includes weight residualinformation and bias residual information indicating a differencebetween a weight and a bias of all or some of filter kernels in all orsome of convolution layers in the up-scaling NN before the weight andthe bias are updated and a weight and a bias of the all or some of thefilter kernels in the all or some of the convolution layers in theup-scaling NN after the weight and the bias are updated when the flaginformation indicates to perform the AI up-scaling by using a filterkernel of a convolution layer in the updated NN.
 9. The electronicdevice of claim 8, wherein the weight residual information and the biasresidual information are information encoded using one of differentialpulse code modulation (DPCM), run-length coding (RLC), and Huffmancoding schemes.
 10. The electronic device of claim 8, wherein the weightresidual information and the bias residual information are informationabout a weight residual and a bias residual generated via modelcompression.
 11. The electronic device of claim 10, wherein the modelcompression comprises at least one of pruning or quantization.
 12. Theelectronic device of claim 1, wherein the NN setting information updatedfor performing the AI up-scaling further includes information about aweight residual and a bias residual obtained by performing frequencytransformation when the flag information indicates to perform the AIup-scaling by using a filter kernel of a convolution layer in theupdated NN, the information about the weight residual and the biasresidual indicating a difference between a weight and a bias of all orsome of filter kernels in all or some of convolution layers in theup-scaling NN before the weight and the bias are updated and a weightand a bias of the all or some of the filter kernels in the all or someof the convolution layers in the up-scaling NN after the weight and thebias are updated.
 13. The electronic device of claim 1, wherein the NNsetting information updated for performing the AI up-scaling furtherincludes information obtained by entropy-encoding a weight and a bias ofall or some of filter kernels in all or some of convolution layers inthe up-scaling NN after the weight and the bias are updated, based oncontext model information regarding a weight and a bias of the all orsome of the filter kernels in the all or some of the convolution layersin the up-scaling NN before the weight and the bias are updated when theflag information indicates to perform the AI up-scaling by using afilter kernel of a convolution layer in the updated NN.
 14. Theelectronic device of claim 1, wherein the NN setting informationincludes NN information updated for performing the AI up-scaling of theat least one second image corresponding to the at least one originalimage via joint training of the up-scaling NN and the down-scaling NNused for the AI down-scaling of the at least one original image.
 15. Amethod of displaying an image by an electronic device configured to usean artificial intelligence (AI), the method comprising: obtaining imagedata generated from performing first encoding on a first image and AIdata related to AI down-scaling of at least one original image relatedto the first image; obtaining a second image corresponding to the firstimage by performing first decoding on the image data; obtaining, basedon the AI data, neural network (NN) setting information for performingAI up-scaling of the second image; identifying whether to perform the AIup-scaling by using a filter kernel of a convolution layer in apredetermined NN or whether to perform the AI up-scaling by using afilter kernel of a convolution layer in a NN updated for performing theAI up-scaling of at least one second image corresponding to at least oneoriginal image via a joint training of an up-scaling NN and adown-scaling NN used for the AI down-scaling of the at least oneoriginal image, the joint training being performed using the at leastoriginal image; and generating a third image by performing the AIup-scaling on the second image via an up-scaling NN operating accordingto the NN setting information based on the identifying.
 16. Anon-transitory computer-readable recording medium having recordedthereon a program for executing the method of claim 15.